SQOOP  : mapred.FileAlreadyExistsException : Output directory
Sometimes when you import data from RDBMS to Hadoop via Sqoop you will see this error. org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://hadoopcluster/user/username/importtable  already exists Solution: $ hdfs dfs -rm -r -skipTrash  hdfs://hadoopcluster/user/username/importtable Reason: When Sqoop is used for Importing data, sqoop creates a temporary file under  home directory and later deletes those files. Sometimes due to some issue,… (0 comment)

gzip / bzip2 can compress individual files. To compress entire folder structure ‘tar’ command comes to our rescue. Compress to .gz  tar -zcf </path/output.gz>  /sourcefolder/path Compress to .bz2 tar -jcf </path/output.bz2> /sourcefolder/path tar options : -z : gzip -c : compress -f : filename -j : bzip2  … (0 comment)

Simple script to covert  regular  date time to unix epoch time format  (UTC) $dat=’01/02/2015′ #If date is passed without time then its assumed as midnight  00:00:00 for that day. $sepoch=$(date “+%s” -d “$dat”) $echo $sepoch will return 1420156800 #If you need for specific datetime, then add time (24 h format) $sepoch=$(date “+%s” -d “$dat  13:15:00”)… (0 comment)

Easy way to split date into  Day, Month, Year  in Linux.  This code was tested using Bash shell. $dat=’13/2/2015′ sday=$(date -d “$dat” ‘+%d’) smonth=$(date -d “$dat” ‘+%m’) sfourdigityear=$(date -d “$dat” ‘+%Y’) stwodigityear=$(date -d “$dat” ‘+%y’) echo $sday  will display   02 echo $smonth  will display   13 echo $sfourdigityear  will display   2015 echo $stwodigityear  will display… (0 comment)