EF Inserts/Updates: Ridiculously Slow

Question

I cracked open the latest version of EF the other day. I had a project where I wanted to insert a lot of data in one hit. The number of records I wanted to insert was about 2,000,000. There are about 8 decimal columns in the table. So, I wrote the code in
EF. I wrote the code with an "Add", and then a "SaveChanges". What was taking an hour and half to write to XML suddenly jumped to an estimated time of 48 hours. (Note: most of the 1 1/2 hours is downloading data from the internet).
So I put an time trace in to see where each action was spending its time. The "Add" (putting the record of the type I want to insert in to the collection) was taking about 300 milliseconds and the "SaveChanges" was taking about 500 milliseconds.
This is simply absurd.
When I changed the code to avoid using EF - i.e. executing a straight INSERT statement against the SQL server database, the time went back down to about 1 1/2 hours. I found that the insert call was only taking about 2-3 milliseconds.
I thought perhaps EF was doing too much at the database level. So, I traced SQL Server. To my surprise, I found that EF was only executing insert statements in the same way I was executing the insert statement. So, why so slow? What's wrong with EF and when
will it be fixed? Is it even possible to use EF?

Answer 1

Several things are important here. First of all: what do you want to do, using which tool,
and how do you implement it.
Concerning the what and which-tool:
several people have already stated that an ORM is not the tool of choice when a bulk insert is what you actually want to do. EF has tons of great features but it’s simply not a solution for every problem.
Then there’s the question of
how you use EF. Two important things that come to mind here are the fact that by default EF tracks all entities that are attached to it, and that a call to savechanges will result in the execution of one insert/update statements for each of the changed
entities.
Let’s assume that EF was applied in this scenario, in the most simple –and naive - way imaginable. We create a context, we add 2M entities, we call SaveChanges.
Then we can take a very long break… Now let’s take a look at what happened here. 2.000.000 entities were attached to the context and it will be tracking all of them for changes – that’s a lot to handle! Once all entities are attached, the SaveChanges method
is called. At this point all modified/new entities are persisted to the database. But EF will actually do this using 2M single insert statement. Ouch.
Now there are two issues here. One is the fact that EF executes only one update/insert statement at a time. You can deal with this using EF extensions that
actually implement the bulk insert you’re looking for. The second issue is the number of entities attached to the context. You can significantly increase performance by lowering the number of attached entities. You could go about this by batching: for the
first set of 500 entities create a new context, add and save the entities; then create a new context for the next 500, and so on.