Redesigning the Collections Framework

Question

Hi!
I'm sort of an experienced Java programmer, in the sense that I program regularly in Java. However, I am not experienced enough to understand the small design specifics of the Collections Framework (or other parts of Javas standard library).
There's been a number of minor things that bugged me about the framework, and all these minor things added up to a big annoyance, after which I've decided to (try to) design and implement my own framework.
The thing is however, that since I don't understand many design specifics about the Collection Framework and the individual collection implementations, I risk coming up short with my own.
(And if you are still reading this, then I thank you for your time, because already now I know that this entry is going to be long. : ) )
Since I'm doing my Collection framework nearly from scratch, I don't have to worry too much about the issue of backwards compatibility (altough I should consider making some parts similar to the collection framework as it is today, and provide a wrapper that implements the original collection interfaces).
I also have certain options of optimizing several of the collections, but then again, there may be very specific design issues concerning performance and usability that the developers of the framework (or other more experienced Java progammers) knew about, that I don't know.
So I'm going to share all of my thoughts here. I hope this will start an interesting discussion : )
(I'm also not going to make a fuss about the source code of my progress. I will happily share it with anyone who is interested. It is probably even neccessary in order for others to understand how I've intended to solve my annoyances (or understand what these annoyances were in the first place). ).
(I've read the "Java Collections API Design FAQ", btw).
Below, I'm going to go through all of the things that I've thought about, and what I've decided to do.
1.
The Collections class is a class that consists only of static utility methods.
Several of them return wrapper classes. However the majority of them work on collections implementing the List interface.
So why weren't they built into the List interface (same goes for methods working only with the Collection interface only, etc)? Several of them can even be implemented more efficiently. For example calling rotate for a LinkedList.
If the LinkedList is circular, using a sentry node connecting the head and tail, rotate is done simply by relocating the sentry node (rotating with one element would require one single operation). The Collections class makes several calls to the reverse method instead (because it lacks access to the internal workings of a LinkedList).
If it were done this way, the Collections class would be much smaller, and contain mostly methods that return wrapped classes.
After thinking about it a while, I think I can answer this question myself. The List interface would become rather bloated, and would force an implementation of methods that the user may not need.
At any rate, I intend to try to do some sort of clean-up. Exactly how, is something I'm still thinking about. Maybe two versions of List interfaces (one "light", and the other advanced), or maybe making the internal fields package private and generally more accessible to other package members, so that methods in other classes can do some optimizations with the additional information.
2.
At one point, I realized that the PriorityQueue didn't support increase\decrease key operations. Of course, elements would need to know where in the backing data structure it was located, and this is implementation specific. However, I was rather dissapointed that this wasn't supported somehow, so i figured out a way to support this anyway, and implemented it.
Basically, I've wrapped the elements in a class that contains this info, and if the element would want to increase its key, it would call a method on the wrapping class it was contained in. It worked fine.
It may cause some overhead, but at least I don't have to re-implement such a datastructure and fiddle around so much with the element-classes just because I want to try something with a PriorityQueue.
I can do the same thing to implement a reusable BinomialHeap, FibonacciHeap, and other datastructures, that usually require that the elements contain some implementation-specific fields and methods.
And this would all become part of the framework.
3.
This one is difficult ot explain.
It basically revolves around the first question in the "Java Collections API Design FAQ".
It has always bothered me that the Collection interface contained methods, that "maybe" would be implemented.
To me it didn't make sense. The Collection should only contain methods that are general for all Collections, such as the contains method. All methods that request, and manipulate the Collection, belonged in other interfaces.
However, I also realized that the whole smart thing about the current Collection interface, is that you can transfer elements from one Collection to another, without needing to know what type of Collection you were transferring from\to.
But I still felt it violated "something" out there, even if it was in the name of convenience. If this convenience was to be provided, it should be done by making a separate wrapper interface with the purpose of grouping the various Collection types together.
If you don't know what I'm trying to say then you might have to see the interfaces I've made.
And while I as at it, I also fiddled with the various method names.
For example, add( int index, E element), I felt it should be insert( int index, E element). This type of minor things caused a lot of confusion for me back then, so I cared enough about this to change it to somthing I thought more appropriate. But I have no idea how appropriate my approach may seem to others. : )
4.
I see an iterator as something that iterates through a collection, and nothing else.
Therefor, it bothered me that the Iterator interface had an optional remove method.
I myself have never needed it, so maybe I just don't know how to appreciate it. How much is it used? If its heavily used, I guess I'm going to have to include it somehow.
5.
A LinkedList doesnt' support random access. But true random access is when you access randomly relative to the first index.
Iterating from the first to the last with a for statement isn't really random access, but it still causes bad performance in the current LinkedList implementation. One would have to use the ListIterator to achieve this.
But even here, if you want a ListIterator that starts in the middle of the list, you still need to traverse the list to reach that point.
So I've come up with LinkedList that remembers the last accessed element using the basic methods get, set, remove etc, and can use it to access elements relatively from it.
Basically, there is now an special interal "ListIterator" that is used to access elements when the basic methods are used. This gives way for several improvements (although that may depend how you look at it).
It introduces some overhead in the form of if-else statemenets, but otherwise, I'm hoping that it will generally outperform the current LinkedList class (when using lists with a large nr of elements).
6.
I've also played around with the ArrayList class.
I've implemented it in a way, that is something like a random-access Deque. This has made it possible to improvement certain methods, like inserting an element\Collection at some index.
Instead of always shifting subsequent element to the right, elements can be shifted left as well. That means that inserting at index 0 only requires a single operation, instead of k * the length of the list.
Again, this intrduces some overhead with if-else statements, but is still better in many cases (again, the List must be large for this to pay off).
7.
I'm also trying to do a hybrid between an ArrayList and a Linked list, hopefully allowing mostly constant insertion, but constant true random access as well. It requires more than twice the memory, since it is backed by both an ArrayList and a LinkedList.
The overhead introduced , and the fact that worst case random access is no better than that of a pure LinkedList (which occurs when elelements are inserted at the same index many times, and you then try to access these elements), may make this class infeasible.
It was mostly the first three points that pushed my over the edge, and made me throw myself at this project.
You're free to comment as much as you like.
If no real discussion starts, thats ok.
Its not like I'm not having fun with this thing on my own : )
I've started from scratch several times because of design problems discovered too late, so if you request to see some of the source code, it is still in the works and I would have to scurry off and add a lot of java-comments as well, to explain code.
Great. Thanks!

Answer 1

This sort of reply has great value : )
Some of them show me that I need to take some other things into consideration. Some of them however, aren't resolved yet, some because I'm probably misunderstanding some of your arguments.
Here goes:
1.
a)
Are you saying that they're were made static, and therefor were implemented in a utility class? Isn't it the other way around? Suppose that I did put them into the List interface, that would mean they don't need to be static anymore, right?
b)
A bloated List interface is a problem. Many of them will however have a default not-so-alwyas-efficient implementation in a abstract base class.
Many of the list-algorithms dump sequential lists in an array, execute the algorithm, and dump the finished list back into a sequential list.
I believe that there are several of them where one of the "dumps" can be avoided.
And even if other algorithms would effectively just be moved around, it wouldn't neccesarily be in the name of performance (some of them cannot really be done better), but in the name of consistency\convenience.
Regarding convenience, I'm getting the message that some may think it more convenient to have these "extra" methods grouped in a utility class. That can be arranged.
But when it comes to consistency with method names (which conacerns usability as well), I felt it is something else entirely.
For example take the two methods fill and replaceAll in the Collections class. They both set specific elements (or all of them) to some value. So they're both related to the set method, but use method names that are very distinguished. For me it make sense to have a method called setAll(...), and overload it. And since the List interface has a set method, I would very much like to group all these related methods together.
Can you follow my idea?
And well, the Collections class would become smaller. If you ask me, it's rather bloated right now, and supports a huge mixed bag of related and unrelated utitlity methods. If we should take this to the extreme, then The Collections class and the Arrays class should be merged.
No, right? That would be hell : )
2,
At a first glance, your route might do the trick. But there's several things here that aren't right
a)
In order to delete an object, you need to know where it is. The only remove method supported by PriorityQueue actually does a linear search. Increase and decrease operations are supposed to be log(n). Doing a linear search would ruin that.
You need a method something like removeAt( int i), where i would be the index in the backing array (assuming you're using an array). The elemeny itself would need to know that int, meaning that it needs an internal int field, even though this field only is required due to the internal workings of PriorityQueue. Every time you want to insert some element, you need to add a field, that really has nothing to with that element from an object-oriented view.
b)
Even if you had such a remove method, using it to increase\decrease key would use up to twice the operations neccesary.
Increasing a key, for example, only requires you to float the element up the heap. You don't need to remove it first, which would require an additional log(n) operations.
3.
I've read the link before, and I agree with them. But I feel that there are other ways to avoid an insane nr of interfaces. I also think I know why I arrive at other design choices.
The Collection interface as it is now, is smart because it can covers a wide range of collection types with add and remove methods. This is useful because you can exchange elements between collections without knowing the type of the collection.
The drawback is of course that not all collection are, e.g modifiable.
What I think the problem is, is that the Collection interface is trying to be two things at once.
On one side, it wants to be the base interface. But on the other side, it wants to cast a wide net over all the collection types.
So what I've done is make a Collection interface that is infact a true base interface, only supporting methods that all collection types have in common.
Then I have a separate interface that tries to support methods for exchanging elements between collections of unknown type.
There isn't neccesarily any performance benefit (actually, it may even introduces some overhead), but in certainly is easier to grasp, for me at least, since it is more logically grouped.
I know, that I'm basically challenging the design choices of Java programmers that have much more experience than me. Hell, they probably already even have considered and rejected what I'm considering now. In that case, I defend myself by mentioning that it isn't described as a possiblity in the FAQ : )
4.
This point is actually related to point 3., becausue if I want the Collection interface to only support common methods, then I can't have an Iterator with a remove method.
But okay....I need to support it somehow. No way around it .
5. 6. & 7.
The message I'm getting here, is that if I implement these changes to LinkedList and ArrayList, then they aren't really LinkedList and ArrayList anymore.
And finally, why do that, when I'm going to do a class that (hopefully) can simulate both anyway?
I hadn't thought of the names as being the problem.
My line of thought was, that okay, you have this arraylist that performs lousy insertion and removal, and so you avoid it.
But occasionally, you need it (don't ask me how often this type of situation arises. Rarely?), and so you would appreciate it if performed "ok". It would still be linear, but would often perform much better (in extreme cases it would be in constant time).
But these improvements would almost certainly change the way one would use LinkedList and ArrayList, and I guess that requires different names for them.
Great input. That I wouldn't have thought of. Thanks.
There is however some comments I should comment:
"And what happens if something is suibsequently inserted or removed between that element and the one you want?"
Then it would perform just like one would expect from a LinkedList. However if that index were closer to the last indexed position, it would be faster. As it is now, LinkedList only chooses either the first index or the last to start the traversal from.
If you're working with a small number of elements, then this is definitely not worth it.
"It sounds to me like this (the hybrid list) is what you really want and indeed all you really need."
You may be right. I felt that since the hybrid list would use twice as much memory, it would not always be the best choice.
I going to think about that one. Thanks.