Application "Icons" no longer showing up in Application Catalog

Question

We have around 500 "Applications" in our application catalog, and make heavy use of the Application Catalog for user self installations. Additionally, we had painstakingly gone through and added "icons" to every application to show up
in Software Center/Application Catalog to make things look "pretty" This had all been working as expected for over a year up until yesterday.
We were rolling out a new App Catalog Website/Web Service (2 servers) to replace our current. I removed the roles from the old one, added to the new. Everything works, software installs work, but no icons show up in the App Catalog. They DO however show
up fine in Software Center. I then reverted the change, putting the role(s) back on the previously working server. No dice, still no icons.
Been playing with it for almost a week, here is what I can discern:
No Icons show up at all, whether custom icons, shell32.dll default icons, or other.
All types do show up in software center after an install is initiated.
The permissions on the "AppIcons" folder are correct (RO to domain users), but the folder itself is empty
No Errors reported on any Roles, or Services (all green)
Modifying an app (removing and re-adding icon) has no effect, still doesn't show up in App Catalog.
Making a NEW app (with new icon) has no effect, still doesn't show up in App Catalog.
Is anyone familiar with the "How do icons get onto the app catalog" process? I assumed they all dumped to the AppIcons folder, but maybe not? Does anyone know what process/log controls this?
Thanks in advance for any help/suggestions, as i want to make sure its not something stupid before blowing $$$ on a MS call.

Answer 1

Wanted to follow up and post a solution to this.
Working with MS support, we were able to identify it as a permissions issue.
Our implementation spans multiple domains, multiple forests, so we have to add in the additional Domains Users with permissions to the "ApplicationCatalog" and "Usericons" folders. I chose to do this via GPO as we have several app
catalog servers, and this seemed "more consistent" to me.
I mistakenly assumed that the privs were identical on both folders when I configured it in GPO. Though very similar, they aren't identical and there are a few extra privs for the SCCM account(s) on the UserIcons folder, which allows it to "write"
to that folder. Simple Fix. Whoops!
Working with MS, I learned a bit more about how the App Catalog works, that I haven't seen written anywhere, so I'll include some of that here. All icons are stored in the SCCM database, and are accessible via the primary site server and WMI. However,
they are "cached" on each application catalog in the appicons folder. This process doesn't happen on a schedule, but rather happens when a user opens the application catalog (on demand). Any icons not retrieved and cached, are cached at that time.
In my case, I had made the GPO change 6 months prior, but no "new" icons had been added, we had been reusing them from app to app (Firefox icon stays the same, and so on). Therefore, it appeared to still be working fine and all icons showed up. In
my case, When I rebuilt the server(s) they needed to "re cache" the icons. When I reverted the app catalog to the old server to revert the change, it also had to "re-cache" the icons as they were deleted when the role was removed, which
was denied.
Hopefully this will help someone in the future if they ever have a similar issue.