Most simple query on Event Hub stream (json) constantly gives Data Conversion Errors

Hello all,
Been playing with ASA in December and didn't have any issues, my queries kept working and outputted the data as needed. However, since January, I created a new demo, where I now constantly get Data Conversion errors. The scenario is described
below, but I have the following questions:
Where can I get detailed information on the data conversion errors? I don't get any point now (not in the operation logs and not in the table storage of my diagnostic storage account)
What could be wrong in my scenario and could be causing these issues
The scenario I have implemented is the following:
My local devices send EventData objects, serialized through Json.Net to an Event Hub with 32 partitions.
I define my query input as Event Hub Stream and define the data as json/utf8. I give it the name TelemetryReadings
Then I write my query as SELECT * FROM TelemetryReadings
In the output, I create an output on blob with CSV/UTF8 encoding
After that, I start the job
The result is an empty blob container (no output written) and tons of data conversion errors in the monitoring graph. What should I do to get this solved?
Thanks
Sam Vanhoutte - CTO Codit - VTS-P BizTalk - Windows Azure Integration: www.integrationcloud.eu

So, apparently the issue was related to the incoming objects, I had. I was sending unsupported data types (boolean and Dictionary). I changed my code to remove these from the json and that worked out well. There was a change that got deployed
that (instead of marking the unsupported fields as null, they were throwing an exception). That's why things worked earlier.
So, it had to do with the limitation that I mentioned in my earlier comment:
https://github.com/Azure/azure-content/blob/master/articles/stream-analytics-limitations.md
Unsupported type conversions result in NULL values
Any event vales with type conversions not supported in the Data Types section of Azure Stream Analytics Query Language
Reference will result in a NULL value. In this preview release no error logging is in place for these conversion exceptions.
I am creating a blog post on this one
Sam Vanhoutte - CTO Codit - VTS-P BizTalk - Windows Azure Integration: www.integrationcloud.eu

Similar Messages

Simple Query Failure? from 2 tables: pull First [DATE] Where [OIL] 0

I am a novice Access 2010 user, and I do not know how to write VBA code. I have written many queries, some built upon others and packaged them into macros, but that's about the extent of my experience, to date.
My problem. I built a query from 2 tables: [AC_PROPERTY] and [AC_PRODUCT]
I have linked the tables by PROPNUM. Designed the query to give me every PROPNUM in the db, then give me the 'First' [DATE] 'Where' [OIL] is >0.
When I run the query with 1 PROPNUM qualified, it works brilliant. Every case I check is working, on a one-PROPNUM-at-a-time basis. As soon as I remove the PROPNUM from criteria, so it will pull all PROPNUMs in the db, it appear to work -- that
is, until you check the data.
For some reason, it's not consistently picking the correct First date where Oil >0. I cannot figure out why it's working on some and not on others. I thought it was a date format problem, but haven't been able to determine or solve the issue.
I have always (historically) figured out my query problems, but this one has me stumped! Help!?
thank you :-)

I have linked the tables by PROPNUM. Designed the query to give me every PROPNUM in the db, then give me the 'First' [DATE] 'Where' [OIL] is >0.
The problem is that FIRST will not always give you the oldest date. Access uses FIRST to get the first record it finds, not the earliest date.
Use MIN to find oldest date.
Build a little, test a little

SSAS source Query, Simple Query with never ending refresh time.

Hello...
A very simple query based on SSAS cube do not give answer. :-(
I start a trace to see the exact mdx statement transmited to SSAS, run it in a query editor, the result is... immediate !
Is there any clue to explain such a behavior ? cube meta data definition ?
Any help is welcome
===========
Version: 2.21.3974.242

hello,
find below the capture of the trace, cannot see any references to rowsets either cellsets...
thank you for your help
select{[Measures].[Fact Count]}on 0,nonempty(crossjoin([Invoice Attributes].[Scenario].[Scenario].allmembers,[Period Date].[Month].[Month].allmembers,[Period Date].[Quarter].[Quarter].allmembers,[Period Date].[Year].[Year].allmembers,[Product Line Techno].[Product
Line Code].[Product Line Code].allmembers),{[Measures].[Fact Count]})properties member_caption,member_unique_name on 1 from(select([Invoice Attributes].[Scenario].&[ACTUALS],[Period Date].[Month].&[20150101],[Product Line Techno].[Product Line Code].&[EB])on
0 from [Revenue Detailed])cell properties value
<PropertyList xmlns="urn:schemas-microsoft-com:xml-analysis">
          <Catalog>OLAP12</Catalog>
          <SspropInitAppName>Microsoft SQL Server Management Studio - Query</SspropInitAppName>
          <LocaleIdentifier>1036</LocaleIdentifier>
          <ClientProcessID>5992</ClientProcessID>
          <Format>Native</Format>
          <AxisFormat>TupleFormat</AxisFormat>
          <Content>SchemaData</Content>
          <Timeout>0</Timeout>
          <DbpropMsmdActivityID>284bd120-34f1-4fac-83cb-e8cf443261ad</DbpropMsmdActivityID>
          <DbpropMsmdRequestID>f11764e9-b092-41ea-aae6-a91763960eae</DbpropMsmdRequestID>
        </PropertyList>

Help understanding simple query trace with 35,000 "query subcube" events

No matter WHAT query I issue against my cube, I'm finding thousands of query subcube events against an unexpected , I assume it's a partition.
Even for the simplest possible query SELECT FROM <CUBE>, which for this cube would return the measure [MRN Distinct Count], a trace reveals 35,000 query subcube events against a partition (?) called "<server>.Medical Home.MedHome.Lab Orders
With Results" which is NOT the partition that holds that [MRN Distinct Count] measure.
I can't figure out why this object is being queried so many times for queries completely unrelated to it. I believe this performance degradation is somewhat new...so I have been trying to think of what may have been added to the cube recently.
I recently added KPIs and calculated sets & members that DO draw from that Lab Orders With Results object...would calculated members or KPI's cause this kind of problem? I thought that they were all pre-calculated @ processing?
Thanks for any insight you can provide
Here's a summary of the profiler trace on the query SELECT FROM <CUBE>
Event Desc
# of Events
ObjectPath
Duration
Query Begin
1
NULL
0
Query Cube Begin
1
<server>.Medical Home.MedHome
0
Query Subcube Verbose
11
<server>.Medical Home.MedHome.Fact Dx 2
189
Query Dimension
7
<server>.Medical Home.Dim Person
0
Query Subcube Verbose
35650
<server>.Medical Home.MedHome.Lab Orders With Results
13856
Serialize Results Begin
1
<server>.Medical Home.MedHome
0
Serialize Results Current
4
<server>.Medical Home.MedHome
0
Query Subcube Verbose
5
<server>.Medical Home.MedHome.Dim Person
0
Serialize Results Current
1
<server>.Medical Home.MedHome
0
Serialize Results End
1
<server>.Medical Home.MedHome
0
Query Cube End
1
<server>.Medical Home.MedHome
40919
Query End
1
NULL
41059
Paul

RedBirdRising - my cube had several dynamic calculated sets in it. As I recall several of those sets included slices on that one particular dimension that was being queried so many times. For me it was sets that compiled populations of people
with particular lab result values, so that lab result partition was queried thousands of times for ANY query, even simple ones like SELECT FROM [CUBE].
To be precise, I actually had calculated sets interacting with calculated measures - for example, one of the offending calculated sets was really this series of calculations:
-CALC SET 1: define calculated set of diabetic patients
-CALC MEASURE 1: define calculated measure for most recent lab test result of a particular lab test - which included a slice on the dimension I found being queried thousands of times.
-DYNAMIC calculated set: FILTER(CALC SET1, CALC MEASURE 1 <= x)
So as soon as I switched that dynamic set to a STATIC set my queries began executing in a reasonable amount of time and those query subcube events were no longer issued in the thousands. I feel like i've heard that FILTER performs poorly but didn't
know of any alternatives to achieve what we needed.
hope this helps...I can't say I know for sure what unforseen impact(s) switching to static sets has had yet. Dynamic seemed like the way to go but the performance hit was just too drastic.
Paul

Best design for Boolean function from simple query

Hello,
what is the most efficient, shorter, readable, simple way to make a boolean function that simply return true or false from a simple query that return 0 or 1 to n records?
Is this solution using a cursor's the best (working...):
   FUNCTION is_date_present (p_date IN DATE)
      RETURN BOOLEAN
   IS
      CURSOR chk_cursor
      IS
         SELECT COUNT (*)
           FROM target_dates
          WHERE target_date = p_date;
      nb   NUMBER := 0;
   BEGIN
      OPEN chk_cursor;
      FETCH chk_cursor
       INTO nb;
      CLOSE chk_cursor;
      IF nb >= 1
      THEN
         RETURN TRUE;
      ELSE
         RETURN FALSE;
      END IF;
   END;Performance, clarity and simplicity are important...
Thanks

Well, I prefer (not tested):
FUNCTION is_date_present (p_date IN DATE)
RETURN BOOLEAN
IS
nb NUMBER := 0;
BEGIN
SELECT COUNT (*)
INTO nb
FROM target_dates
WHERE target_date = p_date;
IF nb >= 1 THEN
RETURN TRUE;
LSE
RETURN FALSE;
IF;
END;Regards,
MiguelWhy count multiple records when you only care if you get at least 1 occurrence? Just wasted cycles.
FUNCTION is_date_present (p_date IN DATE)
RETURN BOOLEAN
IS
nb NUMBER := 0;
BEGIN
SELECT COUNT (*)
INTO nb
FROM DUAL
WHERE EXISTS
      SELECT NULL
      FROM target_dates
      WHERE target_date = p_date
IF nb >= 1 THEN
     RETURN TRUE;
ELSE
     RETURN FALSE;
END IF;
END;Or you could just add a ROWNUM = 1 on to yours, either way.

Help! no response when doing SIMPLE query!

I meet a strange problem when I do a SIMPLE query on a table in Oracle 8.1.7, just like:
SELECT id FROM person WHERE id='12345';
Most of id values are ok to the query, except some values.
Whether I use these values in sqlplus or JDBC, mostly, no response is returned!! And I find the session in Oracle always keeps the status 'Active'. But in few scenario I can get the result quickly.
I don't know the reason. Help me please!
Thanks a lot!
Meyor

I'm not sure if the 'id' column contains something else.
When I shutdown and restart the database, the problem seems to be "solved". If the problem reoccurs again, I will try to use xTRIM functions.
Thanks!
Meyor

How to use peeklock/delete pattern to handle messages in event hub?

Hi,
I have a consumer group with multiple consumers. If one consumer failed to process the message (by some internal reasons. not poison), I want it just leave that message and let other consumers have chance to process it again. previous I use service bus queue/topic,
the peeklock/delete pattern with the dead-letter queue can easily achieve this. Now by some reason I switch event hub,
Is there still some way to achieve this? what is the recommended way to handle this in event hub?
thanks!
Robin

There's no lock and delete concept in EventHubs. Please think EH as a stream of events where you can rewind back in time and start reading from wherever you want. For example as in your case if one of the clients fail to process an event any other client
can start reading from that position where first client failed. Let me know if you have additional questions.
I suggest you to read about EventProcessorHosts for EventHubs. This is probably what you're looking for.
http://azure.microsoft.com/en-us/documentation/articles/service-bus-event-hubs-csharp-ephcs-getstarted/

Event Hubs EventProcessingHost and partitioning over worker roles

Hi,
I have a question after reading the programming guide here:
https://msdn.microsoft.com/en-us/library/azure/dn789972.aspx
My understanding is that if I'm using multiple worker roles on the same event hub, the EventProcessorHost guarantees that a single machine is processing a single partition at any given time. If a checkpoint is made, then the partition might be moved to be
processed on another machine, but in-between checkpoints, all the work for a particular node is done on the same machine.
Is my understanding correct?
Thanks,
Andy

Hi Andy,
Your understanding is partially correct. EPH (event processor host) provides a load balancing context while consuming events from multiple partitions of an event hub. I need to highlight some points below for addressing your possible questions:
* Single host (you called this machine, actually you can run multiple hosts in a single process or i.e. on a single machine) can process multiple partitions.
* Processing a partition at any given time on only one of the hosts is a best effort. There might be situations where 2 hosts end up processing the same event of the same partition when partitions move from one host to another. Please remember that EventHub
doesn't provide lock and delete context, it provides more like a stream instead.
* And also remember that checkpointing is mainly for failing over like in the scenarios as one host crashes/restarts or cannot function as expected so that new host can pick up the partition(s) from the mal-functioned host and continue from wherever it left.
Let me know if you have any questions.

Trouble with a simple Query

Hello, I'll start by saying that I am a noob. Anyways, I am trying to do what I thought would be a simple query to get records that are greater than or equal to the current date: this is my query...
<cfquery name="getUpcoming" datasource="events">
SELECT title, eventDate FROM event WHERE eventDate >= #Now()# ORDER BY eventDate ASC
</cfquery>
It works, sort of, I do get records that are greater than the current date, but any records that are equal to do not show up.
I am assuming that it is looking at time as well, or I am doing it completely wrong. I don't know? Any help would be greatly appreciated.

I didn't use the cfqueryparam as suggested, is there something dangerous about doing it this way?
Nothing dangerous, no. Just "less than ideal" (in a sloppy / lazy sort of way). As I suggested, one should not hard-code dynamic values into the SQL string, one should pass them as parameters. it's just "the way it should be done".
When the DB receives your SQL string (with the dynamic values hard-coded), the DB engine needs to compile the SQL to make an execution plan before executing the query. Any change to the SQL string requires recompilation. However if you pass your parameter as parameters, then the SQL does not need to be recompiled.
It's the same sort of thing as not using global variables unless one has to, despite the fact they're "easier", or duplicating code instead of refactoring code. One should try to write decent code.
Adam

Que about Simple Query Dump.....

Hiiii ABAPers....
Here with i am come again with a que and my que is :
" While debugging a simple query with where clause i am getting dump...but when i set up a break point after 2 or 3 sentenses it is executed fine without any dump "
It has shocked me.... and i m thinking about it by my hand on my head that hows it possible.....
Anyone can help me to put my hand down from the head???
Warm regards,
Nirav Parekh....

HI Nirav
I think you are placing break point in <b>select & end-select</b>
In between select and end-select , if you place any break point ,then there commit statement will trigger,
thats may be the reason ,
so mostly the Programmers are advised to not place any Break points in between Select & end-select.
In your case just place the breakpoint after End-select
or before select
Regards Rk
Message was edited by:
Rk Pasupuleti

Can we use a scheduler to automatically adjust event hub throughput units/

Hi there,
I want to use event hub as our solutions. We have millions of event publishers and consumers. The problem is, we do have peek time which is from 8:00 a.m. - 17:00p.m. At night we may only have few messages. I think event hub bills
against throughput units, so i want to buy more units before 8:00a.m. and reduce the amount at evening. Instead of azure portal, I want to set up a scheduler to help me do this automatically. I didn't find anything on web to tell me how to achieve
this goal.
Please help and advise.
Thanks,
Frederick

No. AQ is not supported by Oracle Streams. User defined and Sys.AnyData are not supported types.
You can create AQ propagation process from source to backup site. But you will need to dequeue both sites simultaniously.
Or you can create schadow table (

Event Hub failover implementation

EventProcessorHost provides a persistent checkpoint storage in blob for failover. But this is only a convenience for failover not failover itself.
Event Hub does not have similar concept as Message Queue paired namespaces for queue redundancy (failover). How would one implement Event Hub redundancy easily?

hi - thanks for your suggestion, but adding "UserProperties" JSON array to the content of the message does not seem to work. When the message is received through IEventProcessor.ProcessEventsAsync()
the Properties dictionary on the EventData instance is still empty.
Please let me clarify what I am trying to achieve here.
I have a plain HTTP client (C++ client) sending messages to the Event Hub via HTTPS. I am trying to achieve similar to
this as I would in a C# client with the service bus client library for event hub:
SomeEventBody body = new SomeEventBody { SomeData = 100 };
EventData data = new EventData(body, serializer) //Object and serializer
// *** I WANT TO SET PROPERTIES ON THE EVENTDATA LIKE THIS ***
data.Properties.Add("Type", "Telemetry_" + DateTime.Now.ToLongTimeString());
await client.SendAsync(data); // Send single message async
When this message is received at the event hub processor, I am able to access EventData.Properties and use the "Type" property in the dictionary.
I want to be able to set the same "Type" property when I send this message from a plain HTTP client, and when the message is received by the event processor I want to be able to read the value out of the dictionary in the same way. I can't though
- because EventData.Properties is always just an empty collection

How to write a simple query.

I have a table where I have data shown below. Now, I want to write a simple query which lists me the project and the count of the distinct effective dates for which data is existant there.
Sample data:
Project Task Effective Date (xx_proj_task_data)
101 T1 01-Jan-2008
101 T1 01-Feb-2008
101 T1 01-Mar-2008
101 T2 01-Jan-2008
101 T2 01-Apr-2008
101 T3 01-Apr-2008
102 T1 01-Jan-2008
102 T1 01-Feb-2008
102 T2 01-Apr-2008
103 T1 01-Jan-2008
103 T1 01-Feb-2008
103 T1 01-Mar-2008
103 T1 01-Apr-2008
103 T2 01-May-2008
103 T3 01-Jun-2008
103 T1 01-Jan-2008
103 T1 01-Aug-2008
103 T2 01-Apr-2008
Output Reqd:
Project Count(Distinct Effective Dates)
101 4
102 3
103 7
I can write a query that says:
select project_id, count(1)
from (select distinct project_id, effective_date
from xx_proj_task_data) x
group by project_id;
But, is there a way I can achieve the same by avoiding the inner Query (x) and just by a simple query ?
Thanks!

Try below query:
select project_id
, count(distinct effective_date)
from xx_proj_task_data
group by project_id;
--venkata

In setting up ICloud on my new Iphone 4s, I lost most of my calendar events in my Outlook 7 calendar. I used to have MobileMe but in the set-up process was told to turn that off which I did but now most of my calendar events have disappeared

In setting up ICloud on my new Iphone 4s, I lost most of my calendar events in my Outlook 7 calendar in my Vista PC. I used to have MobileMe but in the set-up process was told to turn that off, which I did but now most of my calendar events have disappeared after syncing with ICloud. How can I get my missing calendar events back? They are still on my 3Gs Iphone which I have not yet updated to IOS5.
Thanks for any help.
marybpod

Plug your phone into the wall charger for at least 30 minutes...make sure you have a sim card in the phone...then:
Leave the USB cable connected to your computer, but NOT your phone, iTunes running, press & hold the home button while connecting the USB cable to your dock connector, continue holding the home button until you see “Connect to iTunes” on the screen. You may now release the home button. iTunes should now display that it has detected your phone in recovery mode, if not quit and reopen iTunes. If you still don’t see the recovery message repeat these steps again. iTunes will give you the option to restore from a backup or set up as new.
Make sure you have no anti-virus software running or any firewalls...turn all of that stuff off.

Simple Query in Oracle Linked Table in MS Access causes full table scan.

I am running a very simple query in MS ACCESS to a linked Oracle table as follows:
Select *
From EXPRESS_SERVICE_EVENTS --(the linked table name refers to EXPRESS.SERVICE_EVENTS)
Where performed > MyDate()
or
Select *
From EXPRESS_SERVICE_EVENTS --(the linked table name refers to EXPRESS.SERVICE_EVENTS)
Where performed > [Forms]![MyForm]![Date1]
We have over 50 machines and this query runs fine on over half of these, using an Oracle Index on the "performed" field. Running exactly the same thing on the other machines causes a full table scan, therefore ignoring the Index (all machines access the same Access DB).
Strangely, if we write the query as follows:
Select *
From EXPRESS_SERVICE_EVENTS
Where performed > #09/04/2009 08:00#
it works fast everywhere!
Any help on this 'phenominon' would be appreciated.
Things we've done:
Checked regional settings, ODBC driver settings, MS Access settings (as in Tools->Options), we have the latest XP and Office service packs, and re-linked all Access Tables on both the slow and fast machines independantly).

Primarily, thanks gdarling for your reply. This solved our problem.
Just a small note to those who may be using this thread.
Although this might not be the reason, my PC had Oracle 9iR2 installed with Administratiev Tools, where user machines had the same thing installed but using Runtime Installation. For some reason, my PC did not have 'bind date' etc. as an option in the workarounds, but user machines did have this workaround option. Strangely, although I did not have the option, my (ODBC) query was running as expected, but user queries were not.
When we set the workaround checkbox accordingly, the queries then run as expected (fast).
Once again,
Thanks

Most simple query on Event Hub stream (json) constantly gives Data Conversion Errors

Similar Messages

Maybe you are looking for