Fastest way to handle and store a large number of posts in a very short time?

I need to handle a very large number of HTTP posts in a very short period of time. The handling will consist of nothing more than storing the data posted and returning a redirect. The data will be quite small (email, postal code). I don't know exactly how
many posts, but somewhere between 50,000 and 500,000 over the course of a minute.
My plan is to use the traffic manager to distribute the load across several data centers, and to have a website scaled to 10-instances per data center. For storage, I thought that Azure table storage would be the ideal way to handle this, but I'm not sure
if the latency would prevent my app from handling this much data.
Has anyone done anything similar to this and have a suggestion for storing the data? Perhaps buffering everything into memory would be ideal and then batching from there to table storage. I'm starting to load-test the direct to table-storage solution and
am not encouraged.

You are talking about a website with 500,000 posts per minute with re-direction, so you are talking about designing a system that can handle at least 500,000 users? Assuming that not all users are doing posts within a one minute timeframe, then you
are talking about designing a system that can handle millions of users at any one time.
Event hub architecture is completely different from the HTTP post architecture, every device/user/session writes directly to the hub. I was just wondering if that actually work better for you in your situation.
Frank
The site has no session or page displaying. It literally will record a few form values posted from another site and issue a redirect back to that originating site. It is purely for data collection. I'll see if it is possible to write directly to the event hub/service
bus system from a web page. If so, that might work well.

Similar Messages

IMovie: whats the fastest way to import and save all original media?

Good afternoon. Ok I need help please. Is there a way to simply take all the imported original media and just save it all in one fell swoop? Rather than having to import and then process/export clip by clip.
I thought I had a quicker way just grabbing all the original media from the imported camera (right clicking imovie library > show package contents > original media) untill I tried to play one of the original media files - no sound and it was mov file. I assume mp4 is better???
I have 2800 hours of video I need to import off my camera. I don't need to modify them at all, aside from the occasional combining of clips into one movie. So I need the fastest way. Importing and then exporting clip by clip will take me forever. I would prefer to just import and then just drag all 50 clips or so into my movies folder. But as stated above that does not seem to work as expected unless I'm missing a step.
Thank you!

ps Can I take the original media files (mov format) and batch convert them using handbrake or MPEG streamclip? And if so would the resulting videos then have sound? And which is the best format (container codec whatever) to export them to? Thanks!

I've just changed from a PC to a mac and have a large number of downloaded WMA music files (not i-tunes purchases) which I just can't get into my i-tunes library. File converters don't seem to work either. Any ideas?

I've just changed from a PC to a mac and have a large number of downloaded WMA music files which I can't get into i-tunes. When the library was in Windows, i-tunes would convert the files automatically, but this doesn't happen now. I've downloaded a couple of file converters, but these don't seem to work either. Any ideas?

iTunes for Windows (but not iTunes for Mac) can import and convert unprotected WMA tracks. If the tracks are protected by DRM (Digital Rights Management) then it will not accept them.
One option would be to install iTunes on your PC, do the conversion, and then transfer the converted tracks from iTunes on your PC to iTunes on your Mac.

"I am constantly getting a message which says Itunes cannot run because some of it's required files are missing. Please reinstall iTunes." I've reinstalled a number of times and re-booted but it only seems to last a short time. How do I fix it?

I am constantly getting a message which says "iTunes cannot run because some of it's required files are missing. Please reinstall iTunes." I've reinstalled a number of times and re-booted but it only seems to last a short time. How do I fix it?

HI daddio,
Ok some things first I need to know...Did you use Microsoft cleanup utility to clean up all those uninstalls you have done?
I hate Norton and anyone who reads my posts will know that. I have seen way too many infected PC's by Norton users. Did you try the EWIDO and AdAware scan? I see you said you read lots of posts. That is helping folks remove the malware in order to get iTunes and quicktime back to health.

I want to be an OCA and i have a very short time for that

Hello guys this is the same user who wanted to meet some OCA or to-be OCA on stockholm but it didn't work..
Now if anyone can help me i have a very short time to clear my exam i am studying 6 hours a day i have taken off from my work for a month just to clear my test..i have read about 5 chapter now and getting things in my mind but i m not sure if just a book(OCA Oracle Database 11g: SQL Fundamentals I Exam Guide (Exam 1Z0-051)) will be enough or i have to buy some other stuff do we have some workbooks or help meterial or something ...help me guys i have only 3 weeks left ..
I will highly appriciate the help from you guys ..i live in stockholm,sweden if anyone can give me number or something i wana call and ask few things ...kom igen stockholmare kontakta mig och hjälp mig hjälp din svensk bror hehe this message was for swedish helper he he

Thanks for your concern M.r kamran well, i am doing OCA or certification for 11g i thought its the latest thing i am really new in this field and have no idea about this world..as you said that the book i m reading which is ...Oracle Database 11g: SQL Fundamentals I Exam Guide (Exam 1Z0-051)..you said that this is book is enough but the thing is i am not satisfied there are only one lab question and 10 MCQS after each chapter which is i think is not enough to clear the exam...so the problem is i need some material or work book any exam patern or something like that ..where to buy them or how will i get approach of these things? so please if anyone knows where to get those things and how..i will be so much thankfull ..

Why do the "SAVE", "SAVE AS" and "UNDO" functions of InDesign CS6 stop working after a short time?

Why do the "SAVE", "SAVE AS" and "UNDO" functions of InDesign CS6 stop working after a short time?

Normally they don't. You might have some sort of problem on your system, possibly a conflict with other software, or corrupt ID prefs or application. Trashing the prefs is easy, and good place to start: Replace Your Preferences
It will help if you tell us the OS version, whther you've installed the 8.0.1 patch, and how you are trying to access the commands, and what happens when you do. Is this isolated to one file, or does it happen with all files?

(workflow question) - What is the best way to handle audio in a large Premiere project?

Hey all,
This might probably be suitable for any version of Premiere, but just in case, I use CS4 (Master Collection)
I am wrestling in my brain about the best way to handle audio in my project to cut down on the time I am working on it.
This project I just finished was a 10 minute video for a customer shot on miniDV (HVX-200) cut down from 3 hours of tape.
I edited my whole project down to what looked good, and then I decided I needed to clean up all the Audio using Soundbooth, So I had to go in clip by clip, using the Edit in SoundBooth --> Render and Replace method on every clip. I couldn't find a way to batch edit any audio in Soundbooth.
For every clip, I performed similar actions---
1) both tracks of audio were recorded with 2 different microphones (2 mono tracks), so I needed only audio from 1 track - I used SB to cut and paste the good track over the other track.
2) amplified the audio
3) cleaned up the background noise with the noise filter
I am sure there has to be a better workflow option than what I just did (going clip by clip), Can someone give me some advice on how best to handle audio in a situation like this?
Should I have just rendered out new audio for the whole tape I was using, and then edit from that?
Should I have rendered out the audio after I edited the clips into one long track and performed the actions I needed on it? or something entirely different? It was a very slow, tedious process.
Thanks,
Aza

Hi, Aza.
Given that my background is audio and I'm just coming into the brave new world of visual bits and bytes, I would second Hunt's recommendation regarding exporting the entire video's audio as one wav file, working on it, and then reimporting. I do this as one of the last stages, when I know I have the editing done, with an ear towards consistency from beginning to end.
One of the benefits of this approach is that you can manage all audio in the same context. For example, if you want to normalize, compress or limit your audio, doing it a clip at a time will make it difficult for you to match levels consistently or find a compression setting that works smoothly across the board. It's likely that there will instead be subtle or obvious differences between each clip you worked on.
When all your audio is in one file you can, for instance, look at the entire wave form, see that limiting to -6 db would trim off most of the unnecessary peaks, triim it down, and then normalize it all. You may still have to do some tweaking here and there, but it gets you much farther down the road, much more easily.Same goes for reverb, EQ or other effects where you want the same feel throughout the entire video.
Hope this helps,
Chris

Best way to handle mobile accounts with large itunes/photo libraries

What is the best way to handle mobile accounts, but not syncing itunes/photos libraries?
I have a time capsule so I can move itunes and photos libraries for each user if need be.
Thanks!

Hi,
I've done a great deal of work with mobile accounts in Snow Leopard and I'm now having a "play" with Lion. To be honest you have to sit down and think about why you need mobile accounts.
If your user only uses one computer then your safer having a local account backed up by a network Time Machine, this avoids the many many woes that the Servers FileSyncAgent brings to the table.
If your users are going to be accessing multiple computers on the network and leaving the network then a mobile account is good for providing a uniform user experience and access to files etc. However, your users will have to make a choice as to whether they want their iPhoto libraries on one Local machine (backed up by Time Machine) or whether they want their library to be hosted on the server and not part of the Mobile Home Sync schedule (adding ~/Pictures to the excluded items on the home sync settings).
With the latter, users will be able to access their iPhoto libraries on any computer when they are within the network (as it's accessed from the users server home folder).
With the first option the user would have their iPhoto library on one computer (say the laptop they used the most) but then would not be able to access it from other computers they log on to.
iPhoto libraries are a pain, and I'm working hard to come up with a workaround. If your users moved over to using Apeture then you could include the aperture library as part of the home sync thanks to Deeport (http://deepport.net/archives/os-x-portable-home-directories-and-syncing-flaw-wit h-bundles/)
He does suggest that the same would work with IPhoto libraries - but it doesn't for a number of mysterious reasons regarding how the OS recognizes thie iPhoto bundle (it does so differently compared to Apeture).
Hope this helps...

What is the best way to archive and store old mail using Apple Mail?

Hi,
I have a number of older emails that I would like to archive and store for potential future access. Even though I have mails from earlier Lotus Notes accounts and Microsoft Office accounts, I am referring in this case only to apple mail data that I ave 'stored on my mac'.
Any advice would be great. At the moment I am using predominantly gmail, so if things go haywire, rebuilding my mail is relatively easy. That said, I tend to file and save locally important messages.
Cheers,
Mark.

This is from Mail's help:
Archive messages
You can store messages in an Archive mailbox so that you can quickly find them later when you’re ready to take action.
Select one or more messages to archive.If you select a conversation, all messages in the conversation are archived.
Choose Message > Archive.An Archive mailbox is created for each account whose messages you archive. For example, if you archive a message sent to your iCloud account and a message sent to your Yahoo! account, an Archive mailbox is created for each account.Archived messages remain in the Archive mailbox until you delete or move them. Depending on the account type, the Archive mailbox is stored on the mail server.
You can customize the Mail toolbar to include the Archive icon.

Best practice for handling data for a large number of indicators

I'm looking for suggestions or recommendations for how to best handle a UI with a "large" number of indicators. By large I mean enough to make the block diagram quite large and ugly after the data processing for each indicator is added. The data must be "unpacked" and then decoded, e.g., booleans, offset binary bit fields, etc. The indicators are updated once/sec. I am leanding towards a method that worked well for me previously, that is, binding network shared variables to each indicator, then using several sub-vis to process the particular piece of data and write to the appropriate variables.
I was curious what others have done in similar circumstances.
Bill
“A child of five could understand this. Send someone to fetch a child of five.”
― Groucho Marx
Solved!
Go to Solution.

I can certainly feel your pain.
Note that's really what is going on in that png You can see the Action Engine responsible for updating the display to the far right.
In my own defence: the FP concept was presented to the client's customer before they had a person familliar with LabVIEW identified. So I worked it this way from no choice of mine. I knew it would get ugly before I walked in the door and chose to meet the challenge head on anyway. Defer Panel Updates was my very good friend. The sensors these objects represent were constrained to pass info via a single ZigBee network so I had the benefit of fairly low data rates as well but even changing view (Yes there is a display mode that swaps what information is displayed for each sensor) was updated fast enough that the user still got a responsive GUI.
(the GUI did scale poorly though! That is a lot of wires! I was greateful to Jack for the Idea to make align and distribute work on wires)
Jeff

A simple way in sending message on a large number of contacts?

Is there a simple way in sending a large number of contacts? I mean, i'll send my message on a group, i.e. "hello" to "classmates" group. I know it is present during ios4 & 5 version? Why now on ios7? How's that? Some of my contacts are changing new numbers so i can't add them to my inbox.

When a device that is connected to an icloud account has it's contacts deleted, the same contacts at icloud.com are also deleted and then the changes propagate to other computers and devices also connected to the same icloud account. In other words, the contacts are deleted everywhere. Also, any iOS backup to icloud does not include contacts or any of the other Apple data, like calendars, reminders, etc. The only way you can get that data back is if it was on a computer which has been frequently backed up. You restore the data from the backup. (This reference to "backup" is a local backup, it has nothing to do with icloud.)

Best Way to Handle Dynamic Initialization of x number of Objects?

I want to be able to take a x value(integer) that I get from another part of my program and initialize x number of Objects. Best way to handle that?

myObject[] myObjArray = new myObject[x];
for (int i=0; i<x; i++) myObjArray[i] = new myObject("obj#"+i);

What is the best way to import and edit a large amount of footage?

Hello Apple world,
As a newbie to Final Cut 10 I have a question regarding best ways to import footage. I'll give you a quick run down.
I have a 150 gig of gopro footage that I just hit 'import' to recently (from my external HD), it took around 10 hours to process it all.
Its all under one event, but then somewhat organised into dates etc. But it is very slow. I imported it all at once because it was all footage from the last 6 months of my Whistler/states/fiji trip and want to make a movie out of it, using clips from all over the time, not just in chronological order.
I have the latest mac book pro, 4 gig ram,2.4 GHz i7, 750 hard drive, with nothing else on the computer.
Ive read about creating proxy media, after ive imported, but am still somewhat confused as to the benefit. Does it create duplicates? that will fill up my pretty large HD as it is!
Can you suggest any ways that might make my laptop run a bit faster? Should i delete and work directly from external HD etc etc?
Any suggestions will be greatly appreciated!
Thanks in advance
Harlee

ascreenwriter wrote:
Hello,
I've just finished shooting what I am considering to be my directorial masterpiece. Shot it on the Canon 5D (1080p, 24fps), and the footage looks amazing. Now I am ready to start editing and have been using premiere lately, but I have yet to figure out the proper pipeline. I want to know the best way to retain resolution before I delve into this project.
My questions:
1) What is the best way to start a new project and import the footage without having to render whilst editing, so as to retain all resolution and originality of the source footage?
2) What is the best way/ codec/ format to export this same footage once editing is complete so as to retain that crisp 1080p for which the 5D is so recognized?
3) What is the best way/ codec/ format to import and export/ render between premiere and after effects? I am speaking mostly of vfx and color correction. I also have some 30fps footage that I intend to slow down in AE and then import into premiere.
I know this is pretty broad, but as a solo filmmaker I really need someone's guidance. I rarely ever finish my films with the same, crisp look as the footage. I need pipeline help, and really appreciate it!
1. Follow the advice above. Also use the Media Browser to import the footage in case you have spanned media files. Import files with the Media Browser.
2. It largely depends on what you wish to ouput to: Blu-ray, web, etc. This FAQ gives the best answer: What are the best export settings?
3. Use the Replace with Adobe After Effects Composition function.

What is the fastest way to record and write image data?

Hello,
I am new to labview and am using the IMAQ software package to record and write data from a CCD camera to the hard drive. The end goal is a program that records and saves as much data as possible as quickly as possible, for experiments lasting on the scale of days. I have been experimenting with the snap, grab, and sequence methods of recording. To save the image data I was told by NI customer support that TDMS streaming would be the fastest method. I have also been experimenting with the TDMS VIs, however I have found the IMAQ Write File 2 VI to be faster in practice. Am I doing something wrong with the TDMS file structure? Is there a more efficient way to convert IMAQ image data to a dataform that can be written as TDMS? I am posting two of my programs to provide a clearer example of what I am trying to do. The first takes a snap of the image and appends it to a TDMS file. The second is the fastest I have found so far and uses Grab to record the images and the IMAQ Write File VI to save them.
Thanks
Attachments:
Camera Capture (Snap) and stream TDMS.vi ‏24 KB
Camera Capture (Grab) and write image.vi ‏24 KB

Hi
For me it is no surprise that the second VI is faster then the first one, the reason is you can not compare this two VI with each other.
In the first VI you work with TDMS files, in the second with png files. That would be much faster, because TDMS files need a lot of diskspace.
Second point, why do you open, write and close for eacht iteration of the for loop a TDMS file in your first VI? That need really a lot of resources, so it coudn´t be fats as well.
Save your converted pictures in one array, an after the acquisition you can save it in one TDMS file. When you need for each picture a TDMS file you have to know that this need some time to do this.
So now for the architecture of your first VI, please look to the LL Sequence.vifrom the examples (C:\Program Files\National Instruments\LabVIEW 8.6\examples\IMAQ\IMAQ Low Level.llb\LL Sequence.vi), there you can see how an acquisition of a number of images have been to do.
When you like it really fast, you make a producer/consumer architecture (see examples in LabVIEW for that).
So in the first whileloop you acquire the images, write into a queue and in the second, parallel whileloop you read the queue and save the images. (see attachment)
Hope that helps
Kind regards
Beat
NI Germany
Attachments:
ProducerConsumerData.vi ‏10 KB
ProducerConsumerData_LV8.5.vi ‏12 KB

I have a macbook air late 2010 and keep getting a warning that my hard drive is full and to delete things so I purchased a external hard drive (1TB) but I'm just not sure what the correct way to transfer and store files to the external drive is. First MAC

I purchased a macbook air about 5 months ago and lately i keep recieving a warning that the hard drive or disk is full and that I will need to move items to trash in order to free up space on it. I purchased a external hard drive 1 TB but I'm very new to mac and ios and unsure as to how I'm suppose to transfer my files (video, pics, music etc.) from my mac over to the external hard drive in the correct way so I won't end up deleting them permenately or screwing anything up with my mac as far as the applications and programs being able to run. Am I supposed to just copy to the external hard drive and then move to and empty the trash? That's pretty much all I could come up with, but then what if i need or want to access that file again at a later time. Also, I have an i Phone and i pad so with i cloud it automatically syncs all my purcheses and if i delete something from itunes on my mac it deletes from my ipad.
I'd appreciate any kind of input i think im just a little lost here. I have to say i love my Mac and wouldn't trade it for anything (except maybe a macbook pro w/more memory)... Is there something I'm missing???

To free up hard drive space the best bet is to move data files that you don't need access to all the time. Likely candidates are music, video, and photos. Things like word processing and spreadsheet files can also be moved but they tend not to be very large and so don't free up much space. The problem with moving the above mentioned files is that iTunes and iPhoto need to know where the files are stored.
Here's an article explaining how to move the iTunes folder. You can move the iPhoto library using the Finder but there is a slight complication. Start iPhoto, open the Preferences and click on the Advanced tab. The first option is "Copy items to the iPhoto Library". If this option is checked, copy the iPhoto library to your external folder (drag it from the Pictures folder to your external drive) and then delete it from your Picture folder. If this option is not checked, it is a bit more complicated and we'll need to talk a bit.

Fastest way to handle and store a large number of posts in a very short time?

Similar Messages

Maybe you are looking for