Scribd Logo
Upload

Welcome to Scribd!

  • 2 views

    BigData_Setup_guide

    Uploaded by

    簡維萱
    This document provides a step-by-step guide for setting up Maven and Hadoop on Windows, Mac OS, and Linux. It includes instructions for installing Maven and Java, configuring system variables, and building a project using Maven commands. General troubleshooting tips are also provided, such as restarting the IDE or terminal to resolve errors.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd

    BigData_Setup_guide

    Uploaded by

    簡維萱
    2 views3 pages
    This document provides a step-by-step guide for setting up Maven and Hadoop on Windows, Mac OS, and Linux. It includes instructions for installing Maven and Java, configuring system variables, and building a project using Maven commands. General troubleshooting tips are also provided, such as restarting the IDE or terminal to resolve errors.

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    This document provides a step-by-step guide for setting up Maven and Hadoop on Windows, Mac OS, and Linux. It includes instructions for installing Maven and Java, configuring system variables, and building a project using Maven commands. General troubleshooting tips are also provided, such as restarting the IDE or terminal to resolve errors.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    2 views3 pages

    BigData_Setup_guide

    Uploaded by

    簡維萱
    This document provides a step-by-step guide for setting up Maven and Hadoop on Windows, Mac OS, and Linux. It includes instructions for installing Maven and Java, configuring system variables, and building a project using Maven commands. General troubleshooting tips are also provided, such as restarting the IDE or terminal to resolve errors.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    You are on page 1of 3

    Setup Maven + Hadoop in

    General tips
    ● If you get an error, restart your IDE or terminal and try again. This often fixes the
    error.

    Windows
    1. Install Maven & Java

    Go to ​https://maven.apache.org/download.cgi​ and install ​Maven​.

    1) Unpack the archive where you would like to store the binaries, e.g.:

    unzip apache-maven-3.x.y.zip

    2) A directory called "apache-maven-3.x.y" will be created.

    3) Add the bin directory to your PATH, e.g.:

    set PATH="c:\program files\apache-maven-3.x.y\bin";%PATH%

    4) Make sure JAVA_HOME is set to the location of your JDK

    5) Run "mvn --version" to verify that it is correctly installed.

    For installing ​Java 8​ go to


    https://www.oracle.com/nl/java/technologies/javase/javase-jdk8-downloads.html​ (requires
    the creation of an account).

    Verify JAVA_HOME

    1) Open a Command Prompt window (Win⊞ + R, type cmd, hit Enter).


    2) Enter the command ​echo​ %JAVA_HOME% . This should output the path to your
    Java​ installation folder. If it doesn't, your JAVA_HOME variable was not set correctly.

    2. Change local folders

    ● Create a folder named ​C:\hadoop\bin. T ​ wo files must be added which both can be
    found on ​https://github.com/cdarlint/winutils​ (choose Hadoop version 2.6.5). Add the
    following files to ​C:\hadoop\bin​:
    ○ hadoop.dll
    ○ winutils.exe
    ● Add ​hadoop.dll​ to ​C:\Windows\System32

    3. Set System variables

    Go to your Environment Variables.

    !! Only configure ​System variables ​(bottom panel)​, ​NOT User variables (upper panel) !!

    In your System variables, do the following:

    ● Click on ‘New…’:
    1. name: ​HADOOP_HOME
    value: ​C:\hadoop\
    2. name: ​MAVEN_HOME
    value: ​C:\Program Files\apache-maven-3.6.3
    3. name: ​M2_HOME
    value: ​C:\Program Files\apache-maven-3.6.3
    4. name: ​JAVA_HOME
    value: path to your Java JDK (example: ​C:\Program Files\Java\jdk1.8.0_281​)
    ● Click on the ​Path​ variable and choose ‘Edit…’. Then choose ‘New’ and write
    C:\Program Files\apache-maven-3.6.3

    4. Start IDE and build project


    Close all windows and run your IDE or terminal ​as administrator. ​Open the project (see
    https://github.com/schelterlabs/uva-bigdata-course-2021-students.git​) and set your JDK
    under the tab ‘File’ → ‘Project Structure…’ (make sure to set it to the Java 8 JDK you
    installed). Now you can build it using the Maven command:

    mvn -DskipTests install

    To check the code of Exercise 1:

    mvn -Dtest=nl.uva.bigdata.hadoop.exercise1.WordCountLocalTest test


    mvn -Dtest=nl.uva.bigdata.hadoop.exercise1.WordCountClusterTest test

    MAC OS
    ​
    Go to ​https://www.journaldev.com/2348/install-maven-mac-os​ and follow the steps.

    After installing, follow ​step 4​ ​Start IDE and build project ​from the Windows setup (i.e. build
    project and run unit tests). In the case of a Mac OS, you don’t need to run as administrator.

    LINUX
    Go to ​https://linuxize.com/post/how-to-install-apache-maven-on-ubuntu-18-04/​ and follow the
    steps.

    In the case that you get errors due to a wrong JDK version, you have to switch between
    versions. For more information see
    https://blogs.igalia.com/dpino/2011/10/13/configuring-different-jdks-with-alternatives/​.

    After installing, follow ​step 4 Start IDE and build project​ from the Windows setup (i.e. build
    project and run unit tests). In the case of a Linux OS, you don’t need to run as administrator.

    You might also like