Nutch

To contribute a patch, follow these instructions (note that installing Hub is not strictly required, but is recommended). Pull requests run Apache Yetus test-patch for automated checks (style, reporting). See Basic Precommit and [Usage Introduction][yetus-usage]. CI uses Java 17. To run test-patch locally (e.g. before opening a PR): You must [configure the nutch-site.xml][runnutch] before running. Make sure you have added http.agent.name and plugin.folders properties. The plugin.folders normally points to /build/plugins . Now create a Java Application Configuration, choose org.apache.nutch.crawl.Injector, add two paths as arguments: first the crawldb directory, second the URL directory where the injector can read urls. Then run your configuration. First install the [IvyIDEA Plugin][ivyidea]. Then run ant eclipse . This creates the .classpath and .project files so Intellij can import the project. In Intellij IDEA, select File New Project from Existing Sources. Select the nutch home directory and click "Open". On the "Import Project" screen select the "Import project from external model" radio button and select "Eclipse". Click "Create". On the next screen the "Eclipse projects directory" should be already set to the nutch folder. Leave the "Create module files near .classpath files" radio button selected. Click "Next" on the next screens. On the project SDK screen select Java 17 (or newer) and click "Create". N.B. On Mac with homebrew openjdk, use the directory under libexec : /libexec/openjdk.jdk/Contents/Home . Once the project is imported, you will see a popup saying "Ant build scripts found", "Frameworks detected - IvyIDEA Framework detected". Click "Import". If you don't get the pop-up, go through the steps again as this happens from time to time. There is another Ant popup that asks you to configure the project. Do NOT click "Configure". To import the code-style: Intellij IDEA Preferences Editor Code Style Java. For the Scheme dropdown select "Project". Click the gear icon and select "Import Scheme" "Eclipse XML file". Select the eclipse-format.xml file and click "Open". On the next screen check the "Current Scheme" checkbox and hit OK. Running in Intellij Open Run/Debug Configurations Select "+" to create a new configuration and select "Application" For "Main Class" enter a class with a main function (e.g. org.apache.nutch.indexer.IndexingJob) For "Program Arguments" add the arguments needed for the class. You can get these by running the crawl executable for your job. Use full-qualified paths (e.g. crawldb and segments paths plus -deleteGone) For "Working Directory" enter your nutch runtime/local path Select "Modify options" "Modify Classpath" and add the config directory for that Working Directory (e.g. runtime/local/conf) Select "Modify options" "Add VM Options" and add the VM options from running the crawl executable (e.g. -Xmx4096m -Dhadoop.log.dir=... etc.) Note : You will need to manually trigger a build through ANT to get latest updated changes when running, because the ant build system is separate from the Intellij one.