Use Fuse to mount HDFS

2011-09-13  来源:本站原创  分类:Internet  人气:215 

Premise has been installed hadoop cluster

1 install fuse
Here you can compile and install or yum install

yum install fuse fuse-libs fuse-devel


2 Installing ant

Use yum to install

yum install ant


3 Add the system configuration

vi / etc / profile, depending on the actual situation to set their own

export OS_ARCH=amd64
export OS_BIT=64
export JAVA_HOME=/usr/java/jdk1.6.0_26
export CLASSPATH=,:$JAVA_HOME/lib
export HADOOP_HOME=/usr/local/hadoop-0.20.2
export LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/$OS_ARCH/server:$HADOOP_HOME/c++/Linux-$OS_ARCH-$OS_BIT/lib:/usr/local/lib:/usr/lib
source /etc/profile


4. Compile

cd $HADOOP_HOME
ant compile-c++-libhdfs -Dlibhdfs=1 -Dcompile.c++=1
ln -s c++/Linux-$OS_ARCH-$OS_BIT/lib build/libhdfs
ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1

5. <br /> Start test after a hadoop cluster,

mkdir /mnt/dfs
cd $HADOOP_HOME/build/contrib/fuse-dfs
./fuse_dfs_wrapper.sh dfs://namenode:9000 /mnt/dfs/
ls /mnt/dfs/
相关文章
  • Use Fuse to mount HDFS 2011-09-13

    Premise has been installed hadoop cluster 1 install fuse Here you can compile and install or yum install yum install fuse fuse-libs fuse-devel 2 Installing ant Use yum to install yum install ant 3 Add the system configuration vi / etc / profile, depe

  • LINUX mount an NTFS file system 2010-05-11

    wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.1-1.el5.rf.x86_64.rpm rpm -i rpmforge-release-0.5.1-1.el5.rf.*.rpm rpm --import http://dag.wieers.com/rpm/packages/RPM-GPG-KEY.dag.txt rpm -K rpmforge-release-0.5.1-1.el5.rf.*.rpm rpm -

  • Unable to mount NTFS formatted partition: FATAL: Module fuse not found. 2010-09-07

    Question: Can not mount the CentOS 5.3 on NTFS partition, the following message appears: # Mount-t ntfs-3g / dev/sdb1 / mnt / usb / FATAL: Module fuse not found. ntfs-3g-mount: fuse device is missing, try 'modprobe fuse' as root This is because the s

  • CentOS mount NTFS Device 2010-03-23

    mount NTFS device: yum install fuse fuse-ntfs-3g dkms dkms-fuse edit / etc / fstab as follows. To mount read-only: /dev/sda1 /mymnt/win ntfs-3g ro,umask=0222,defaults 0 0 mount / mymnt / win unmount NTFS device: unmount / mymnt / win Reference: http:

  • Linux mount NTFS disk 2010-04-27

    Today, the company's servers to do maintenance, you need to backup some data back. Ntfs with the past is a mobile hard drive, to the room, I suddenly thought of the original company's servers also can not be directly linked to the hard disk in ntfs f

  • Linux mount NTFS drives can read and write 2010-07-15

    NTFS-3G is an open source software to support Linux, FreeBSD, Mac OS X, NetBSD, Haiku operating system read and write NTFS formatted partitions. It can fast and secure operation of Windows XP, Windows Server 2003, Windows 2000 and Windows Vista file

  • Linux can read and write NTFS drives mount 2010-07-15

    NTFS-3G is an open source software to support Linux, FreeBSD, Mac OS X, NetBSD, Haiku operating system read and write NTFS formatted partitions. It can fast and secure operation of Windows XP, Windows Server 2003, Windows 2000 and Windows Vista file

  • Red Hat Enterprise Linux 5 using the experience (1) NTFS reel mount and can read and write set 2010-08-07

    Transfer: http://hi.baidu.com/56908268/blog/item/3143c3957bcaf541d0135e60.html How to mount an NTFS formatted hard drive to linux system? Need to download a patch to support NTFS. 1, NTFS mount reel A test with your kernel version and CPU type # Unam

  • (Transfer) centos mount ntfs HDD 2010-08-27

    1, first make sure fuse, CentOS 5.5 with a fuse, you can use rpm-qa | grep fuse to see if installed. [Root @ localhost ntfs-3g-2010.5.22] # rpm-qa | grep fuse fuse-2.7.4-8.el5 Fuse already installed If you do not, do [Root @ localhost ntfs-3g-2010.5.

  • kvm virtual machine system mount an external hard drive (img file) 2010-11-26

    The previous article about how to create a linux kvm virtualization-based system. The significance of the virtual machine is to install the full use of resources, to do cloud computing cluster. As with clustering, to relate to the virtual machine cop

  • Mount NTFS partition in RHEL6 and let it read and write 2011-05-07

    How to mount NTFS partitions and allowed to read and write inside the first step is to compile the kernel. First download the latest version of ntfs-3g, the current version is The latest stable version is ntfs-3g_ntfsprogs-2011.4.12, released on Apri

  • Linux mount ntfs partition 2011-02-20

    As the installation of linux and xp dual system, but they can only support linux under windows fat and fat32 file systems. So to the original xp ntfs partition data to use in linux, you have the ntfs partition where things move to the fat partition a

  • centos (linux) to mount ntfs file 2011-03-05

    In his recent useless desktop installed above centos, ready to install some software to do the above distribution of the cluster test. It very fast loading system can be fitted with a mounted NTFS file not found. Google today, the specific operation

  • centos ntfs partition format hard drive mount 2011-06-14

    Original address http://longwang617.iteye.com/blog/749061 1, first make sure the fuse, CentOS 5.5 with a fuse, you can use rpm-qa | grep fuse to see if installed. [Root @ localhost ntfs-3g-2010.5.22] # rpm-qa | grep fuse fuse-2.7.4-8.el5 Already inst

  • 用Python编写一个简单的FUSE文件系统的教程 2014-05-26

    这篇文章主要介绍了用Python编写一个简单的FUSE文件系统的教程,对于数据的备份很有帮助,需要的朋友可以参考下 如果你是我的长期读者,那么你应该知道我在寻找一个完美备份程序,最后我写了一个基于bup的我自己的加密层. 在写encbup的时候,我对仅仅恢复一个文件就必须要下载整个巨大的档案文件的做法不甚满意,但仍然希望能将EncFS和 rdiff-backup一起使用来实现可远程挂载.加密.去重.版本化备份的功能. 再次试用obnam 后(??乱痪洌核?故锹?某銎?,我注意到了它有一个moun

  • 大数据开发之深入HDFS 2015-04-18

    当数据集的大小超过一台独立的物理计算机的存储能力时,就有必要对它进行分区(partition)并存储到若干台单独的计算机上.管理网络中跨多台计算机存储的文件系统称为分布式文件系统(distributedfilesystem).该系统架构于网络之上,势必会引入网络编程的复杂性,因此分布式文件系统比普通磁盘文件系统更为复杂.例如,使文件系统能够容忍节点故障且不丢失任何数据,就是一个极大的挑战. Hadoop有一个称为HDFS的分布式系统,即HadoopDistributedFilesystem.在非

  • 用Python写一个FUSE(用户态文件系统)文件系统 2014-11-26

    如果你是我的长期读者,那么你应该知道我在寻找一个完美备份程序,最后我写了一个基于bup的我自己的加密层. 在写encbup的时候,我对仅仅恢复一个文件就必须要下载整个巨大的档案文件的做法不甚满意,但仍然希望能将EncFS和rdiff-backup一起使用来实现可远程挂载.加密.去重.版本化备份的功能. 再次试用obnam后(??乱痪洌核?故锹?某銎?,我注意到了它有一个mount命令.深入研究后,我发现了fuse-python和fusepy,感觉用Python写一个FUSE文件系统应该挺简单的.

  • HDFS写入和读取流程 2012-02-21

    一.HDFS HDFS全称是Hadoop Distributed System.HDFS是为以流的方式存取大文件而设计的.适用于几百MB,GB以及TB,并写一次读多次的场合.而对于低延时数据访问.大量小文件.同时写和任意的文件修改,则并不是十分适合. 目前HDFS支持的使用接口除了Java的还有,Thrift.C.FUSE.WebDAV.HTTP等.HDFS是以block-sized chunk组织其文件内容的,默认的block大小为64MB,对于不足64MB的文件,其会占用一个block,但实

  • Hadoop2.2.0用HDFS NFS Gateway挂载HDFS文件系统 2014-05-09

    现在项目需要将CloudStack的存储进行分布式处理,现在Hadoop集群已经搭建完毕,可以通过配置CloudSatck的基础架构,将用nfs的方式添加主存或者二级存储.注意,这个特性是Hadoop2.2.0开始支持的,而且不够完善.比如只支持顺序读,也就是说不能随机读写.因此通过NFS网关方式挂载HDFS文件系统,并不能把它完全当作本地文件使用.比较适合的场景是文件上传和文件下载.文件浏览等(感觉百度网盘就是这个场景).通过这项技术,我们倒是可以把它当作一个云端的FTP服务器使用了,我们可以

  • HDFS Federation设计动机与基本原理 2015-01-29

    HDFS Federation是Hadoop最新发布版本Hadoop-0.23.0中为解决HDFS单点故障而提出的namenode水平扩展方案.该方案允许HDFS创建多个namespace以提高集群的扩展性和隔离性.本篇文章主要介绍了HDFS Federation的设计动机和基本原理. 1. 当前HDFS概况 1.1 当前HDFS架构 当前HDFS包含两层结构: (1) Namespace 管理目录,文件和数据块.它支持常见的文件系统操作,如创建文件,修改文件,删除文件等. (2) Block