Flume Installation

Flume :-

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.

1) Download "apache-flume-1.4.0-bin.tar.gz" from apache org.

2) Extract it and place it ino /usr/local directory.

3) Add the below lines into flume.conf.properties file (make a new file from template)

a1.sources = s1

a1.channels = c1

a1.sinks = sk1

# For each one of the sources, the type is defined

a1.sources.s1.type = exec

a1.sources.s1.command = tail -F /home/Fission/Downloads/flume-demo.txt

# Each channel's type is defined.

a1.channels.c1.type = memory

a1.channels.c1.capacity = 1000

a1.channels.c1.transactioCapacity=100

#Linking source with channel and channel with sink.

a1.sources.s1.channels = c1

a1.sinks.sk1.channel = c1

4) Command to run flume agent/master :

go to flume directory then run this command : bin/flume-ng agent --conf conf --conf-file conf/flume-conf.properties --name a1 -Dflume.root.logger=INFO,console