Zfs list on solaris express 11 always reads from disk (metadata not cached)

Question

Hello All,
I am migrating from OpenSolaris 2009.11 to SolarisExpress 11.
I noticed that "zfs list" takes longer than usual, and is not instant. I then discovered via a combination of arcstat.pl and iostat -xnc 2 that every time a list command is issued, there are disk reads. This leads me to believe that some metadata is not getting cached.
This is not the case in OpenSolaris where repeated "zfs list" do not cause disk reads.
Has anyone observed this, and do you know of any solution?
This is on an IDLE sustem with 48 GB of RAM - with plenty of free memory.

Answer 1

Hi Steve,
Great info again. I am still new to dtrace, particularly navigating probes and etc. I've seen that navigation tree before.
I would like to start by answering your questions:
Q) Have you implemented any ARC tuning to limit the ARC?
-> No out of the box config
Q) Are you running short on memory? (the memstat above should tell you)
-> Definetelly not. I have 48 GB ram, ARC grows to about 38 GB and then stops growing. I can reproduce problem at boot at will with only 8GB used. Nothing is getting aged out of the ARC at that time. Only those metadata reads never get stored.
Q) Are any of your fileystems over 70% full?
-> No. I am curious, what changes when this happens? Particularly in regards to ARC - perhaps another discussion, I don't want to distract this subject.
Q) Have you altered what data is/is not cached? ($ zfs get primarycache)
-> No - everything should be cached. I also have recently added l2cache (80GB). The metadata is not cached there neither.
I am not yet familiar with dtrace processing capabilities, thus I had to parse output via perl. Notice how each execution has the exact same number of misses. This is due to the fact that these particular datablocks (metadata blocks) are not cached in the arc at all:
:~/dtrace# perl -MData::Dumper -e 'while (<>) {if (/.+(:arc-hit|:arc-miss).*/) { $h{$1}+=1}} print Dumper \%h' ^C
:~/dtrace# ./zfs_list.d -c 'zfs list' |perl -MData::Dumper -e 'while (<>) {if (/.+(:arc-hit|:arc-miss).*/) { $h{$1}+=1}} print Dumper \%h'
dtrace: script './zfs_list.d' matched 4828 probes
dtrace: pid 11021 has exited
$VAR1 = {
          ':arc-hit' => 2,
          ':arc-miss' => 192
:~/dtrace# ./zfs_list.d -c 'zfs list' |perl -MData::Dumper -e 'while (<>) {if (/.+(:arc-hit|:arc-miss).*/) { $h{$1}+=1}} print Dumper \%h'
dtrace: script './zfs_list.d' matched 4828 probes
dtrace: pid 11026 has exited
$VAR1 = {
          ':arc-hit' => 1,
          ':arc-miss' => 192
:~/dtrace# ./zfs_list.d -c 'zfs list' |perl -MData::Dumper -e 'while (<>) {if (/.+(:arc-hit|:arc-miss).*/) { $h{$1}+=1}} print Dumper \%h'
dtrace: script './zfs_list.d' matched 4828 probes
dtrace: pid 11031 has exited
$VAR1 = {
          ':arc-hit' => 12,
          ':arc-miss' => 192
:~/dtrace# ./zfs_list.d -c 'zfs list' |perl -MData::Dumper -e 'while (<>) {if (/.+(:arc-hit|:arc-miss).*/) { $h{$1}+=1}} print Dumper \%h'
dtrace: script './zfs_list.d' matched 4828 probes
dtrace: pid 11036 has exited
$VAR1 = {
          ':arc-hit' => 4,
          ':arc-miss' => 192
:~/dtrace# ./zfs_list.d -c 'zfs list' |perl -MData::Dumper -e 'while (<>) {if (/.+(:arc-hit|:arc-miss).*/) { $h{$1}+=1}} print Dumper \%h'
dtrace: script './zfs_list.d' matched 4828 probes
dtrace: pid 11041 has exited
$VAR1 = {
          ':arc-hit' => 27,
          ':arc-miss' => 192
:~/dtrace# I presume next steps would be to perform stack analysis on which blocks are been not cached. I don't know how to do this ... I am guessing this is a mid-function probe? "| arc_read_nolock:arc-miss" I don't know how to access it's parameters.
FYI, here's an example of a cache miss in my zfs list:
  0  -> arc_read                             
  0    -> arc_read_nolock                    
  0      -> spa_guid                         
  0      <- spa_guid                         
  0      -> buf_hash_find                    
  0        -> buf_hash                       
  0        <- buf_hash                       
  0      <- buf_hash_find                    
  0      -> add_reference                    
  0      <- add_reference                    
  0      -> buf_cons                         
  0        -> arc_space_consume              
  0        <- arc_space_consume              
  0      <- buf_cons                         
  0      -> arc_get_data_buf                 
  0        -> arc_adapt                      
  0          -> arc_reclaim_needed           
  0          <- arc_reclaim_needed           
  0        <- arc_adapt                      
  0        -> arc_evict_needed               
  0          -> arc_reclaim_needed           
  0          <- arc_reclaim_needed           
  0        <- arc_evict_needed               
  0        -> zio_buf_alloc                  
  0        <- zio_buf_alloc                  
  0        -> arc_space_consume              
  0        <- arc_space_consume              
  0      <- arc_get_data_buf                 
  0      -> arc_access                       
  0       | arc_access:new_state-mfu         
  0        -> arc_change_state               
  0        <- arc_change_state               
  0      <- arc_access                       
  0     | arc_read_nolock:arc-miss           
  0     | arc_read_nolock:l2arc-miss 
  0      -> zio_read                         
  0        -> zio_create                     
  0          -> zio_add_child                
  0          <- zio_add_child                
  0        <- zio_create                     
  0      <- zio_read                         
  0      -> zio_nowait                       
  0        -> zio_unique_parent              
  0          -> zio_walk_parents             
  0          <- zio_walk_parents             
  0          -> zio_walk_parents             
  0          <- zio_walk_parents             
  0        <- zio_unique_parent              
  0        -> zio_execute                    
  0          -> zio_read_bp_init             
  0            -> zio_buf_alloc              
  0            <- zio_buf_alloc              
  0            -> zio_push_transform         
  0            <- zio_push_transform         
  0          <- zio_read_bp_init             
  0          -> zio_ready                    
  0            -> zio_wait_for_children      
  0            <- zio_wait_for_children      
  0            -> zio_wait_for_children      
  0            <- zio_wait_for_children      
  0            -> zio_walk_parents           
  0            <- zio_walk_parents           
  0            -> zio_walk_parents           
  0            <- zio_walk_parents           
  0            -> zio_notify_parent          
  0            <- zio_notify_parent          
  0          <- zio_ready                    
  0          -> zio_taskq_member             
  0          <- zio_taskq_member             
  0          -> zio_vdev_io_start            
  0            -> spa_config_enter           
  0            <- spa_config_enter           
  0            -> vdev_mirror_io_start       
  0              -> vdev_mirror_map_alloc    
  0                -> spa_get_random         
  0                <- spa_get_random         
  0                -> vdev_lookup_top        
  0                <- vdev_lookup_top        
  0              <- vdev_mirror_map_alloc    
  0              -> vdev_mirror_child_select 
  0                -> vdev_readable          
  0                  -> vdev_is_dead         
  0                  <- vdev_is_dead         
  0                <- vdev_readable          
  0                -> vdev_dtl_contains      
  0                <- vdev_dtl_contains      
  0              <- vdev_mirror_child_select 
  0              -> zio_vdev_child_io        
  0                -> zio_create             
  0                  -> zio_add_child        
  0                  <- zio_add_child        
  0                <- zio_create             
  0              <- zio_vdev_child_io        
  0              -> zio_nowait               
  0                -> zio_execute            
  0                  -> zio_vdev_io_start    
  0                    -> spa_syncing_txg    
  0                    <- spa_syncing_txg    
  0                    -> zio_buf_alloc      
  0                    <- zio_buf_alloc      
  0                    -> zio_push_transform 
  0                    <- zio_push_transform 
  0                    -> vdev_mirror_io_start
  0                      -> vdev_mirror_map_alloc
  0                      <- vdev_mirror_map_alloc
  0                      -> vdev_mirror_child_select
  0                        -> vdev_readable  
  0                          -> vdev_is_dead 
  0                          <- vdev_is_dead 
  0                        <- vdev_readable  
  0                        -> vdev_dtl_contains
  0                        <- vdev_dtl_contains
  0                      <- vdev_mirror_child_select
  0                      -> zio_vdev_child_io
  0                        -> zio_create     
  0                          -> zio_add_child
  0                          <- zio_add_child
  0                        <- zio_create     
  0                      <- zio_vdev_child_io
  0                      -> zio_nowait       
  0                        -> zio_execute    
  0                          -> zio_vdev_io_start
  0                            -> vdev_cache_read
  0                              -> vdev_cache_allocate
  0                              <- vdev_cache_allocate
  0                            <- vdev_cache_read
  0                            -> vdev_queue_io
  0                              -> vdev_queue_io_add
  0                              <- vdev_queue_io_add
  0                              -> vdev_queue_io_to_issue
  0                                -> vdev_queue_io_remove
  0                                <- vdev_queue_io_remove
  0                              <- vdev_queue_io_to_issue
  0                            <- vdev_queue_io
  0                            -> vdev_accessible
  0                              -> vdev_is_dead
  0                              <- vdev_is_dead
  0                            <- vdev_accessible
  0                            -> vdev_disk_io_start
  0                            <- vdev_disk_io_start
  0                          <- zio_vdev_io_start
  0                        <- zio_execute    
  0                      <- zio_nowait       
  0                    <- vdev_mirror_io_start
  0                  <- zio_vdev_io_start    
  0                  -> zio_vdev_io_done     
  0                    -> zio_wait_for_children
  0                    <- zio_wait_for_children
  0                  <- zio_vdev_io_done     
  0                <- zio_execute            
  0              <- zio_nowait               
  0            <- vdev_mirror_io_start       
  0          <- zio_vdev_io_start            
  0          -> zio_vdev_io_done             
  0            -> zio_wait_for_children      
  0            <- zio_wait_for_children      
  0          <- zio_vdev_io_done             
  0        <- zio_execute                    
  0      <- zio_nowait                       
  0    <- arc_read_nolock                    
  0  <- arc_read                              I've compared the output of a single-non cached metadata read, to a single read from filesystem by running dd and read from a file that is not in the cache. The only difference in the stack is that the non-cached reads are missing:
  0                                -> vdev_queue_offset_compare
  0                                <- vdev_queue_offset_compare This is called in "-> vdev_queue_io_to_issue ". But I don't think this is relevant, perhaps related to metadata vs file data read.
What do you think should be next?