Pig Latin: Load multiple files from a date range (part of the directory structure)

Question

I have the following scenario-

Pig version used 0.70

Sample HDFS directory structure:

/user/training/test/20100810/<data files>

/user/training/test/20100811/<data files>

/user/training/test/20100812/<data files>

/user/training/test/20100813/<data files>

/user/training/test/20100814/<data files>

As you can see in the paths listed above, one of the directory names is a date stamp.

Problem: I want to load files from a date range say from 20100810 to 20100813.

I can pass the 'from' and 'to' of the date range as parameters to the Pig script but how do I make use of these parameters in the LOAD statement. I am able to do the following

temp = LOAD '/user/training/test/{20100810,20100811,20100812}' USING SomeLoader() AS (...);

The following works with hadoop:

hadoop fs -ls /user/training/test/{20100810..20100813}

But it fails when I try the same with LOAD inside the pig script. How do I make use of the parameters passed to the Pig script to load data from a date range?

Error log follows:

Backend error message during job submission

-------------------------------------------

org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Unable to create input splits for: hdfs://<ServerName>.com/user/training/test/{20100810..20100813}

        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:269)

        at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:858)

        at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:875)

        at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)

        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:793)

        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:752)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:396)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1062)

        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:752)

        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:726)

        at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)

        at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)

        at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)

        at java.lang.Thread.run(Thread.java:619)

Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input Pattern hdfs://<ServerName>.com/user/training/test/{20100810..20100813} matches 0 files

        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:231)

        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)

        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:248)

        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:258)

        ... 14 more

Pig Stack Trace

---------------

ERROR 2997: Unable to recreate exception from backend error: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Unable to create input splits for: hdfs://<ServerName>.com/user/training/test/{20100810..20100813}

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias test

        at org.apache.pig.PigServer.openIterator(PigServer.java:521)

        at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:544)

        at org.apache.pig.tools.pigscript.pars

Answer 1

One common way to solve your problem is to simply use Pig parameters (which is a good way to make your script more reusable anyway). 

Different mechanisms to define parameters that can be referenced in a Pig Latin script are:

• Parameters can be defined as command-line arguments; each parameter is passed to Pig as a separate argument using -param switches at script execution time

• Parameters can be defined in a parameter file that's passed to Pig using the -param_file command-line argument when the script is executed

• Parameters can be defined inside Pig Latin scripts using the "%declare" and "%default" preprocessor statements

shell:

pig -f script.pig -param input=/user/training/test/{20100810..20100812}

Or

pig -f script.pig -param input=’/user/training/test/{20100810..20100812}’

script.pig:

temp = LOAD '$input' USING SomeLoader() AS (...);

If you want to know more about Apache Pig, then do check out this awesome video tutorial: