Very low arcsize problem.

For weeks now I've been scrambling to find a cause for our newly created Solaris 11 NAS slowness issue. The server has two SSD drives for Sol 11 in a mirrored rpool and four SATA drives in a mirror'd pool for data with many NFS mounts. It has 16GB of memory and a recently added Samsung SSD log drive for the data pool.
The server uses IP over InfiniBand as a storage network between a ESXi node hosting approx 15 NFS attached vm's.
We have great performance for approximately three days, then for no apparent reason, IO crawls. Going from upper 300MB/s range on the VM's down to 10-15MB/s. When looking on the Solaris side, I see the same throughput change eliminating the IB network or something with ESXi. When this state occurs, system memory goes from (in good state) 2.5G free to ~13GB free. The only solution to this state seems to be a reboot of Solaris.
After doing some research, I found some users out there(not on this forum) that had seen the same issue. They had found it revolved around the arc dropping to a very low state for no good reason. I looked when my server was slow and sure enough, my arc size was under 100M and I had over 13GB free. They had also found that the arc_no_grow flag was getting set to 1 and was not getting set back to 0 to allow the ARC to take back the memory from perceived memory pressure situation which they or myself have nothing running on the servers to create a memory pressure situation.
I have yet to find a fix for this or maybe even a work around. I attempted to set "zfs_arc_min" to 4GB, this change took effect but just like before, after a few days the arc dropped, only difference is this time its seems to be staying up around ~200MB and only dipping in the 90MB's very rarely allowing us to stay up at the time of this writing at over seven days. Since putting this in, IO doesn't bottom out like before, and allows us to still run about 50% of a normal IO.
I dont have dedupe or compression on anywhere. The arc_min set to 4GB is allowing us to at least use the NAS but its a shame that 13GB is free at all times that could be used for cache and we are not getting the great performance we know we can get for a few days after a reboot.
If anyone has any information on this or have experienced this before, please share.

Thank you for your response and suggestions. The server memory is never affected by any process consuming memory or needing memory.I've yet to setup any backups or zfs replication. I was using a perl script called "arcstat.pl" to watch the arc this last time it happened. It literally went a arcsize of 10G down to 574M in under 20 minutes. At that time I was watching all the process memory consumption with prstat and top. I am going to setup sar as you suggested to help better watch things, thank you for that suggestion.
time     read     miss     miss% dmis dm% pmis     pm% mmis mm% arcsz     c
13:30:40     232     18     7     8     3     10     50     0     1     10G     11G
13:40:40     2.4K     717     30     9     0     707     90     7     0     8.5G     8.5G
13:50:40     4.4K     2.2K     48     169     8     2.0K     81     21     1     574M     4.0G
14:00:40     3.7K     2.2K     58     263     19     1.9K     80     27     4     234M     4.0G
14:10:40     3.9K     2.3K     59     255     19     2.0K     79     25     4     160M     4.0G
14:20:40     4.7K     2.8K     60     307     20     2.5K     79     29     3     211M     4.0G
14:30:40     3.7K     2.1K     55     188     13     1.9K     81     24     2     131M     4.0G
14:40:40     2.5K     1.5K     61     224     25     1.3K     80     32     7     127M     4.0G
14:50:40     2.8K     1.7K     59     238     23     1.4K     81     41     8     168M     4.0G
15:00:40     7.1K     4.2K     58     359     15     3.8K     80     50     3     185M     4.0G
15:10:40     6.3K     3.9K     61     409     21     3.5K     78     42     4     238M     4.0G
This was on 3/2/12 and the server is still up, arcz is still floating from 90M to at most I've seen looking through the logs ~700M. Before I set the zfs_arc_min, this looked the same with the exception of two things, arcz floated around the 40-50MB's, "c" around the same and IO was terrible.
Here is some detailed arc and memory info if this helps:
uptime
3:46pm up 7 day(s), 16:26, 1 user, load average: 0.06, 0.05, 0.05
/arc_summary.pl
System Memory:
Physical RAM: 16342 MB
Free Memory : 12863 MB
LotsFree: 255 MB
ZFS Tunables (/etc/system):
set zfs:zfs_arc_min = 4294967296
ARC Size:
Current Size: 154 MB (arcsize)
Target Size (Adaptive): 4096 MB (c)
Min Size (Hard Limit): 4096 MB (zfs_arc_min)
Max Size (Hard Limit): 15318 MB (zfs_arc_max)
ARC Size Breakdown:
Most Recently Used Cache Size: 6% 256 MB (p)
Most Frequently Used Cache Size: 93% 3839 MB (c-p)
ARC Efficency:
Cache Access Total: 335961590
Cache Hit Ratio: 52% 176807570 [Defined State for buffer]
Cache Miss Ratio: 47% 159154020 [Undefined State for Buffer]
REAL Hit Ratio: 77% 261160422 [MRU/MFU Hits Only]
Data Demand Efficiency: 65%
Data Prefetch Efficiency: 13%
CACHE HITS BY CACHE LIST:
Anon: --% Counter Rolled.
Most Recently Used: 46% 82783131 (mru) [ Return Customer ]
Most Frequently Used: 100% 178377291 (mfu) [ Frequent Customer ]
Most Recently Used Ghost: 0% 0 (mru_ghost) [ Return Customer Evicted, Now Back ]
Most Frequently Used Ghost: 0% 0 (mfu_ghost) [ Frequent Customer Evicted, Now Back ]
CACHE HITS BY DATA TYPE:
Demand Data: 58% 103054838
Prefetch Data: 8% 14640256
Demand Metadata: 33% 58571606
Prefetch Metadata: 0% 540870
CACHE MISSES BY DATA TYPE:
Demand Data: 33% 53709385
Prefetch Data: 58% 92853092
Demand Metadata: 5% 9285319
Prefetch Metadata: 2% 3306224
::memstatPage Summary Pages MB %Tot
Kernel 839647 3279 20%
ZFS File Data 16542 64 0%
Anon 28776 112 1%
Exec and libs 1514 5 0%
Page cache 7189 28 0%
Free (cachelist) 9940 38 0%
Free (freelist) 3280164 12813 78%
Total 4183772 16342
Physical 4183771 16342
SIZE RES STATE TIME CPU COMMAND
39M 22M sleep 0:14 0.00% fmd
20M 12M sleep 0:09 0.00% svc.startd
20M 19M sleep 0:56 0.00% svc.configd
18M 5192K sleep 0:09 0.00% sshd
18M 5188K sleep 0:21 0.00% sshd
18M 4704K sleep 0:03 0.00% sshd
18M 4660K sleep 0:01 0.00% sshd
15M 12M sleep 0:02 0.00% arcstat.pl
15M 3512K sleep 0:00 0.00% rad
14M 4680K sleep 0:00 0.00% kcfd
14M 3004K sleep 0:00 0.00% sshd
13M 4748K sleep 0:20 0.00% nscd
13M 4680K sleep 0:00 0.00% syseventd
13M 6776K sleep 0:11 0.00% devfsadm
12M 1928K sleep 0:19 0.00% nfsmapid
11M 3300K sleep 0:05 0.00% inetd
11M 2628K sleep 0:03 0.00% mountd
11M 1276K sleep 0:00 0.00% net-physical
11M 2060K sleep 0:00 0.00% syslogd
11M 2772K sleep 0:00 0.00% picld
10M 6556K sleep 0:00 0.00% bash
10M 2208K sleep 0:00 0.00% bash
10M 1968K sleep 0:00 0.00% asr-notify
10M 2100K sleep 0:00 0.00% smtp-notify

Similar Messages

5gr - please wait - very low battery

ive ipod video 30 giga (5gr)
and i hade the very low battery problem
i tried to charge it from my cpu (usb)
and i triend from the wall too
i have the ipod 18 mounths - so i tought the battery is dead... so i bought a new battery
when i connected the battery, the ipod turn on and working well (apart from this problem - the ipod allways tought he are connecting to some charger)
after 30 min i think.... he turned off and show the same problem "please wait - very low battery "
so i dont know what to do...
i cant do nothing...
only reset...
no HD mode...
nohing... there is no battery (and i cant turn it off - the massage is allways showen)
i dont have right now wall charger
one more thing -
while it connected to the cpu i heared it beeping and some massege was appear... i cant remmember what... it was longest then "please wait - very low battery"
and then it disappeared and the battey low massege was back
help plz
thanks
ipod video 30 gb (5gr) Windows XP Pro

its not work...
i did the auto test to the ipod
(reset - and then menu+previus)
i connected it the the charger...
now i disconnect the ipod... and the ipod steel think that he connect to some charger. if i want to connect it to the compuet, i need to reset the ipod.
after the ipod is connecting to the computer... or playing music...
he back to "very low battery"
the problem is not with the battery.. i test it with 2 difrent battery...

How do I fix my ipod classic that says very low batter after I have charged it and the problem persists?

I have an Ipod video 30g that says Please wait very low battery. I charged it for about a day and the problem persists. I've tried using some solutions on the internet and none of them worked including: holding the menu and select, holding select and play to put it in disk mode with no avail. Is there anyone who can help me!!!

Have you read this post written by another forum member?
The Sad iPod icon.
However, as your iPod was purchased on Boxing Day, why not get it serviced under the warranty?
You can arrange online service here.
Service request.
Or if an Apple store is near you, take it there and have them check your iPod.
You can make an appointment by using this link.
Genius Bar Appointments.

Volume on iphone is very low, this is a common problem, Apple Needs to FIX!

I have a new iphone 3GS that I got at the end of July, 2009. Since then i've been having problems with the volume! I called support because i had this problem along with my screen turning blank and freezing up. Support said to take the case off and this might help. well it did help the screen issue (which is kind of dumb when you think about it because the case is supposed to protect the phone and instead it caused it to malfunction!!,wierd), but the volume issue never really changed. I am hearing impaired and I do wear hearing aids, BUT....I have ALWAYS been able to use cell phones that have good loud volume, as i need to remove my aids to use the phone. My LG phone had BEAUTIFUL volume, never had a problem!
So, then I started to do some research and saw this is a very common problem! everyone I know that has an iphone has this problem, and all these people are normal hearing people! So then i decided to just get some really good, superduper headphones, because the headphones that came with the phone were ok, but i had to hold them in my ears.
The other day i bought the new Bose in-ear headphones with superior sound quality and at $130 bucks, they better be and the are compatable with the iphone. But.....again, while actually talking to someone on the phone the sound, even with the headphones and all volumes max'd, was still very low!! and i know its not the headphones because listening to the ipod is fantastic! it must be the phone!!!
I even used the "accessibility" setting that is on the phone and selected the mono audio feature which pushes both L&R sounds through one channel, and still doesnt help!
I even called support back today and one guy said i had to "restore" my phone by plugging it into my computer (not really sure what he means) but we got disconnected and the other guy said that i should bring my phone into the apple store. Both tech guys denied that there are lots of calls or complaints about this....but as you all know from reading the many posts, that this is a very common problem!!!
I think Apple should do the right thing and recall these phones or at least fix them or SOMETHING!!!! this volume issue is internally, because when i make a call and an automated voice comes on, it works fine (majority of the time) but as soon as a live voice comes on the phone, the volume drops! there must be some type of automatic voice adjuster that is just not working properly. I mean i love my phone!!! But....i cant use it!!! And now, im going to have to return my Bose headphones and try to find something else.
Can anyone really help me?????

stewa348 wrote:
I agree with you about the volume on the phone itself. (I have 3Gs). It needs to be louder. I also wear hearing aids. However I do think there may be something wrong with your phone because when I connect my earbuds (Klipsh) I get very good volume through them. In fact at full volume it's way too loud. I would push for a replacement phone.
FWIW, my daughter (who also wears hearing aids) has trouble hearing her iphone. It appears to have more to do with the speaker not being lined up with the hearing aid's input. Her audiologist pointed out that some current hearing aids support bluetooth, I think we will get those next time we replace her hearing aids.

Could Very Low Disk Space Cause problems?

Hi, my 600 GB Hard Drive only had around 5 GB of space left . I purchased a new 1 TB Hard Drive in order to delete media so that I could gain the space back. I've since gained back 200 GB of Space but Could very low disk space cause issues? What kind of issues could occur? Thanks
Message was edited by: Stuart Lawrence

There are any number of reasons for occasional beachballs to appear. They will appear when some programs are involved in heavy disk I/O which tends to be processor dependent and causes the program to suspend all other activity until the disk activity has completed. This is more often the case when writing to the disk. Beachballs may also appear if the OS has to use disk I/O and is slowed down by an over-filled and/or fragmented hard drive. The fuller a drive gets the greater the likelihood that fragmentation will become a more noticeable problem. Beachballs can appear if you are low on physical memory so applications must resort to disk-based virtual memory which is considerably slower and tends to cause the computer to become slow or unresponsive for a period of time. It can also cause disk-thrashing. Beachballs can occur if the disk has become corrupted causing applications to have difficulty reading from or writing to the disk.
It is possible to repair a drive's directory, and it's even possible to identify corrupted files, but it's not possible to repair corrupted files nor repair a corrupted drive. If a drive develops corrupted files either the files will need to be replaced or deleted. If a drive develops soft sector errors these can be repaired by erasing the drive (reformatting.) If a drive develops hard sector errors these may or may not be repairable, but if they are repairable it requires zeroing the data during a disk formatting operation. If the disk directory becomes corrupted that can be repaired using Disk Utility or one of the various third-party utilities such as Disk Warrior. Disk Utility can repair many but not all directory corruption problems. What DU cannot fix Disk Warrior usually can.
You can start repairing a drive as follows;
Repairing the Hard Drive and Permissions
Boot from your OS X Installer disc. After the installer loads select your language and click on the Continue button. When the menu bar appears select Disk Utility from the Installer menu (Utilities menu for Tiger and Leopard.) After DU loads select your hard drive entry (mfgr.'s ID and drive size) from the the left side list. In the DU status area you will see an entry for the S.M.A.R.T. status of the hard drive. If it does not say "Verified" then the hard drive is failing or failed. (SMART status is not reported on external Firewire or USB drives.) If the drive is "Verified" then select your OS X volume from the list on the left (sub-entry below the drive entry,) click on the First Aid tab, then click on the Repair Disk button. If DU reports any errors that have been fixed, then re-run Repair Disk until no errors are reported. If no errors are reported click on the Repair Permissions button. Wait until the operation completes, then quit DU and return to the installer. Now restart normally.
If DU reports errors it cannot fix, then you will need Disk Warrior and/or Tech Tool Pro to repair the drive. If you don't have either of them or if neither of them can fix the drive, then you will need to reformat the drive and reinstall OS X.
This applies to your startup volume. No need to repair permissions on any but a startup volume.

[linux 3.0] Problem with sound : very low.

Since I upgraded to linux 3.0, I noticed that sound is very very low. Even if gnome sound properties tells me it is set at 100%, I nearly cannot hear it at all.
I got this in dmesg | tail :
[fred@fredo-arch ~]$ dmesg | tail
[ 10.943927] NET: Registered protocol family 31
[ 10.943932] Bluetooth: HCI device and connection manager initialized
[ 10.943940] Bluetooth: HCI socket layer initialized
[ 10.943943] Bluetooth: L2CAP socket layer initialized
[ 10.944041] Bluetooth: SCO socket layer initialized
[ 10.981755] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[ 19.925726] hda-intel: IRQ timing workaround is activated for card #1. Suggest a bigger bdl_pos_adj.
[ 20.203365] eth0: no IPv6 routers present
[ 22.618181] EXT4-fs (sda4): re-mounted. Opts: commit=0
[ 22.755787] EXT4-fs (sda5): re-mounted. Opts: commit=0
If I plug an headset in the front on my PC, I got sound with the right power. Weird. Any tips ?

same here. almost no sound at all (unless you have superman ears).
00:1b.0 Audio device: Intel Corporation N10/ICH 7 Family High Definition Audio Controller (rev 01)
sounds like a alsa problem?
Edit:
This fixed it:
I closed kde's mixer then I opened alsamixer. I selected Master channel and pressed M on my keyboard to unmute.
then I did "/etc/rc.d/alsa stop" to save the sound volume status. and then "/etc/rc.d/alsa start" and I opened kmix again.
Last edited by hussam (2011-07-24 20:48:55)

HT1222 Please Help.. When I update to 7.0.6 I remark a very low battery lifetime I does not complete 2 hours.. I update to 7.1 to get rid of this problem but unforeately the problem still.. Could you please guide what to do?

Please Help.. When I update to 7.0.6 I remark a very low battery lifetime I does not complete 2 hours.. I update to 7.1 to get rid of this problem but unforeately the problem still.. Could you please guide what to do?

Restore as New (no backup)... then Re-Sync your Content.
Make sure you have the Latest Version of iTunes (v11.1.5) Installed on your computer
iTunes free download from www.itunes.com/download

HT4623 My ringer and music volume is very low!! The problem just started today can someone help?

Can someone one help me, I just started having problems with my volume. My music and my ringtone are very low!

As a first step, do a hard reset by holding down Sleep/Wake and Home buttons until you see the Apple logo on the screen. If a red power slider appears on the screen, just ignore it and keep holding the buttons down until you see the Apple logo.

My volume on the Iphone 3GS is very low is it an equipment problem, setting problem or update problem?

My volume on the iPhone 3GS is very low. Is it an equipment problem, settings problem or update problem? And, is there a fix other than having Apple fix it?

Have you had a look here to make sure it is set up how you want it...
iPhone User Guide
http://manuals.info.apple.com/en_US/iphone_user_guide.pdf

TS3899 Upgrade to ios6 got the following problems in my ipad2 . Very low download speed. Can not open my apple mail in my iPad.

Upgrade to iOS 6 encountered the following problems in my ipad2.
(1) very low download speed
(2) I can not open my apple gmail in my iPad.
(3) I can't download new apps from Internet.

Have tried your suggestion of toggling but unfortunately that has not solved the problem - but many thanks for the suggestion.
Regards
Sailorfrank

OK, here is my problem...i leave my ipod classic charging for 2 days and it still has the 'Please wait, very low battery" sign! what should i do? Please answer!

OK, here is my problem...i leave my ipod classic charging for 2 days and it still has the 'Please wait, very low battery" sign! what should i do? Please answer!

When you connect the iPod to the adapter or PC, you should see either of these signs on the iPod Screen .
If you just get
It may mean that your Adapter is not giving enough juice, because
the adapter or the cable is defective or
your bottom dock connector is dirty with lint or dirt.
Have a nice day!

HT1849 hello i downloaded an album on my iPad yesterday but the sound volume is very low. Can you help me to fix the problem ?

hello i downloaded an album on my iPad yesterday but the sound volume is very low. Can someone help me to fix the problem ?

Hello Kishore1022,
It sounds like you have a new album downloaded and the tracks all have a very low volume. I recommend checking to see if Sound Check is enabled in the Music settings.
iPad User Guide
http://help.apple.com/ipad/7/#/iPad99f37037
Go to Settings > Music to set options for Music, including:
Sound Check (to normalize the volume level of your audio content)
You can also try downloading the purchases onto a computer, with either of these articles depending on your version of the operating system:
iTunes 11 for Windows: Download previous purchases from the iTunes Store
http://support.apple.com/kb/PH12491
Or
iTunes 11 for Mac: Download previous purchases from the iTunes Store
http://support.apple.com/kb/PH12283
Thank you for using Apple Support Communities.
Cheers,
Sterling
Thank you for using Apple Support Communities.
All the best,
Sterling

After update 6.0.1 i have problem with soun it become very low pls solve thise by update another can i restoer to 6 another time

after update to 6.0.1 the sound very low how solve thise problem can i restor to ios 6 another time you solve problem battery by update 6.01 but you make problem with rining sound pls solve solve problem

http://support.apple.com/kb/TS2802

When I make an outgoing call the recipient cannot hear me very well as though the volume was turned down very low. However when I activate the speakerphone the recipient can hear me loud and clear. I transferred my SIM Card into a new device Same problem

When I make an outgoing call the recipient cannot hear me very well as though the volume was turned down very low. However when I activate the speakerphone the recipient can hear me loud and clear. I transferred my SIM Card into a new device only to be subjected to the exact same problem. I have reset the phone clearing all the data and updated to the most recent software version available from itunes; and still no improvements. The results of my switching to a new phone would indicate a software issue.
Does anyone out there have this problem and how can it be fixed?
Thanks,
Teresa

Try removing the screen protector that is covering the microphone.
Scratch that; wrong answer. I thought you meant you had trouble hearing.

HT1435 hi there i have this problem with my ipod on the second picture which says "Pease Wait Very Low Battery" but it will not charge at all or turn on, sync with computer anybody any ideas.

Hi there I have an ipod classic which will not charge and keeps saying "Please Wait Very Low Battery" can anbody help me its just sad to see it go.

Normally it takes 2 to 4 hours to fully charge an iPod Battery, using a USB2.0 port or wall charger.
Even if the battery is dead, you can have it replaced at Apple Store for about US$60 or at any 3rd Party Ipod Service Center for less.

Very low arcsize problem.

Similar Messages

Maybe you are looking for