Add even one song to iPhone (manual sync) and now iPhone erases rest of music library?

Question

Many thanks for your attention.
This has been happening more and more - especially with more recent iOS and iTunes updates.
I long have kept many Movies and much Music on my iPod or (now) iPhone.
In the past, whenever I wanted to add a new song to the iPhone, I would just drag and drop the song into it from my desktop.
Almost all of my music I converted over from CDs I own. Same with the movies.
I almost never buy music or movies from the iTunes store.
I have always had iTunes Sync set to Manual.
Until a couple years ago, no problems.
Now, sometimes I drag one film over or add one new song - and the rest of my music library (and all the movies) on my iPhone are seemingly automatically erased. Poof - they're all gone!
It doesn't happen each time - sometimes the song or songs are copied over and my iPhone music library is added to - as I planned.
But sometimes, I perform the same step - and the new song is added - but all (over 4000) of the songs already in library, all the playlists, etc., vanish.
It takes a while to add over 4000 songs to the iPhone (again and again). I had read over last couple years of bugs that would be fixed, etc., but it seems to be much worse now, not better.
Except - the few songs I once downloaded from the iTunes store. Those appear - ready to play yet  - but ready to be downloaded again.
I have tried several things to combat this - deleted Pref files, etc. But there is no pref file on a Mac for the iPhone. Only for iTunes.
I should say I do not keep an iTunes Library on my computer. It is mostly empty (to save HD space). As my iPhone is 64GB - that would be 64GB of needless space taken up - when I only listed to music or watch movies via my iPhone. Up until about two years ago - this was never an issue though, since back to the first iPod. And I have always had my iTunes set to Manual Sync.
I do not use iCloud to backup my music.
I would just manually add a song or a movie to iPhone via my desktop.
Any suggestions to stop this from repeatedly happening welcome. My thanks in advance for any help.

Answer 1

Many thanks for reply and suggestions, Lawrence.
If a corrupt music database on iPhone - why was this happening on iPods (in the past)  as well? 
I have done complete Restores and Re-installs of iOS in the past. But this problem keeps popping up.
But - will try your suggestion.
It seems to be connected to Apple moving to iCloud and people backing up their music there. Also, many (or most) people also keep all their music on their Mac. I don't do either. iPhone always recognizes the few songs or films I downloaded from Store (they are also erased - but I have the joy of the option to download them again).
But - most of my music and movies come from my own CD collection and my own DVD collection - not from iTunes Store.
Your Playlist of All Music suggestion. If I understand right, I would have to load all music to iTunes first (some 40GB?)
Then Sync.
But I don't want to keep all 40GB+ of music perpetually on Mac.