Hakuna MapData! » Blog Archive » What I did when assigning META region in HBase cluster took ages…
rss

What I did when assigning META region in HBase cluster took ages…

| Posted in Troubleshooting |

Today, I faced the issue of inability to assign META region to a regionserver when starting HBase cluster. Basically ROOT region was assigned correctly and quickly (just a matter of seconds), where META region was permanently in transition state. Obviously and unfortunately, it resulted in HBase cluster inaccessibility.

Since it have never encountered such situation before, I did not know best-to-use solution.

What I finally do was to remove /hbase/-ROOT- and /hbase/.META. directories (actually, I moved them to other localization in case of the total disaster) and restart a cluster. Then the cluster started successfully and both ROOT and META regions were assigned correctly. So far so good!

However when I checked the consistency of the HBase cluster with hbase hbck, it looked to be very inconsistent…

1
2
3
4
5
6
7
8
9
10
11
12
13
sudo -u hdfs hbase hbck -fixAssignments  
 
ERROR: Region { meta => null, hdfs => hdfs://hadoop-master/hbase/some-table/ffecc045335a93dc0185a610b20ceae6, deployed =>  } on HDFS, but not listed in META or deployed on any region server
...
Summary:
  -ROOT- is okay.
    Number of regions: 1
    Deployed on:  hadoop-slave2,60020,1348681480029
  .META. is okay.
    Number of regions: 1
    Deployed on:  hadoop-slave1,60020,1348681480009
843 inconsistencies detected.
Status: INCONSISTENT

I got 843 inconsistencies due to errors like Region { meta => null, hdfs => hdfs://hadoop-master/hbase/some-table/ffecc045335a93dc0185a610b20ceae6, deployed => } on HDFS, but not listed in META or deployed on any region server.

Unfortunately hbase hbck -fixAssignments did not fix this (since there were rather missing region entries in META, not incorrectly assigned regions). Luckily hbase hbck have more problem-specific parameters that I have learnt by reading HBase in Action by N.Dimiduk and A.Khurana. One of them is -repairHoles and it solved my problem.

1
2
3
4
sudo -u hdfs hbase hbck -repairHoles
...
0 inconsistencies detected.
Status: OK

Actually, -repairHoles is a shortcut for -fixAssignments -fixMeta -fixHdfsHoles -fixHdfsOrphans. You may get the list of all options by typing:

1
hbase hbck --help

Please let me know, if you find better way to solve this problem.

No related posts found.

VN:F [1.9.20_1166]

Rate this post!

Rating: 5.0/5 (4 votes cast)
What I did when assigning META region in HBase cluster took ages..., 5.0 out of 5 based on 4 ratings

Comments

comments