Запускаю job в gobblin'e
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
48.
49.
50.
51.
52.
53.
54.
55.
56.
57.
58.
59.
60.
61.
62.
63.
64.
65.
66.
67.
68.
69.
70.
71.
72.
73.
74.
75.
76.
77.
78.
79.
80.
81.
Результат журналов логов vim gobblin-current.log
2019-03-06 14:01:11 UTC INFO [SchedulerService STARTING] org.quartz.impl.StdSchedulerFactory 1327 - Quartz scheduler 'LocalJobScheduler' initialized from specified file: '/opt/gobblin/gobblin-dist/conf/quartz.properties'
2019-03-06 14:01:11 UTC INFO [SchedulerService STARTING] org.quartz.impl.StdSchedulerFactory 1331 - Quartz scheduler version: 2.2.3
2019-03-06 14:01:11 UTC INFO [SchedulerService STARTING] org.quartz.core.QuartzScheduler 575 - Scheduler LocalJobScheduler_$_NON_CLUSTERED started.
2019-03-06 14:01:12 UTC WARN [JobScheduler STARTING] org.apache.hadoop.util.NativeCodeLoader 62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-03-06 14:01:13 UTC INFO [JobScheduler STARTING] org.apache.gobblin.scheduler.JobScheduler 507 - Scheduling configured jobs
2019-03-06 14:01:14 UTC INFO [JobScheduler STARTING] org.apache.gobblin.scheduler.JobScheduler 527 - Loaded 1 job configurations
2019-03-06 14:01:14 UTC INFO [JobScheduler-0] org.apache.gobblin.util.ClustersNames 74 - no default cluster mapping found
2019-03-06 14:01:14 UTC WARN [JobScheduler-0] org.apache.gobblin.runtime.AbstractJobLauncher 223 - Creating a job specific SharedResourcesBroker. Objects will only be shared at the job level.
2019-03-06 14:01:17 UTC INFO [JobScheduler-0] org.reflections.Reflections 229 - Reflections took 2597 ms to scan 48 urls, producing 809 keys and 2849 values
2019-03-06 14:01:18 UTC ERROR [JobScheduler-0] org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner 629 - Failed to run job GobblinKafkaQuickStart
org.apache.gobblin.runtime.JobException: Failed to run job GobblinKafkaQuickStart
at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:437)
at org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner.run(JobScheduler.java:627)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Failed to create job launcher: java.lang.RuntimeException: Failed to create FsDatasetStateStore with factory
at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:158)
at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:107)
at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:85)
at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:68)
at org.apache.gobblin.scheduler.JobScheduler.buildJobLauncher(JobScheduler.java:442)
at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:435)
... 4 more
Caused by: java.lang.RuntimeException: Failed to create FsDatasetStateStore with factory
at org.apache.gobblin.runtime.FsDatasetStateStoreFactory.createStateStore(FsDatasetStateStoreFactory.java:31)
at org.apache.gobblin.runtime.JobContext.createStateStore(JobContext.java:200)
at org.apache.gobblin.runtime.JobContext.<init>(JobContext.java:147)
at org.apache.gobblin.runtime.AbstractJobLauncher.<init>(AbstractJobLauncher.java:183)
at org.apache.gobblin.runtime.local.LocalJobLauncher.<init>(LocalJobLauncher.java:81)
at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:144)
... 9 more
Caused by: java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme: instance-1.europe-west1-b.c.absolute-axis-232916.internal
at org.apache.gobblin.runtime.FsDatasetStateStore.createStateStore(FsDatasetStateStore.java:133)
at org.apache.gobblin.runtime.FsDatasetStateStoreFactory.createStateStore(FsDatasetStateStoreFactory.java:29)
... 14 more
Caused by: java.io.IOException: No FileSystem for scheme: instance-1.europe-west1-b.c.absolute-axis-232916.internal
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2304)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2311)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
at org.apache.gobblin.runtime.FsDatasetStateStore.createStateStore(FsDatasetStateStore.java:107)
... 15 more
Файл настройки джоба.
kafka-gobblin-hdfs.pull
###################### job configuration file ######################
job.name=GobblinKafkaQuickStart
job.group=GobblinKafka
job.description=Gobblin quick start job for Kafka
job.lock.enabled=false
kafka.brokers=0.0.0.0:6667
source.class=gobblin.source.extractor.extract.kafka.KafkaSimpleSource
extract.namespace=gobblin.extract.kafka
writer.builder.class=gobblin.writer.SimpleDataWriterBuilder
writer.file.path.type=tablename
writer.destination.type=HDFS
writer.output.format=txt
data.publisher.type=gobblin.publisher.BaseDataPublisher
mr.job.max.mappers=1
metrics.reporting.file.enabled=true
metrics.log.dir=/gobblin-kafka/metrics
metrics.reporting.file.suffix=txt
bootstrap.with.offset=earliest
fs.uri=instance-1.europe-west1-b.c.absolute-axis-232916.internal:8020;
writer.fs.uri=instance-1.europe-west1-b.c.absolute-axis-232916.internal:8020;
state.store.fs.uri=instance-1.europe-west1-b.c.absolute-axis-232916.internal:8020;
mr.job.root.dir=/opt/gobblin/gobblin-dist/job_work/working
state.store.dir=/opt/gobblin/gobblin-dist/job_work/state-store
task.data.root.dir=/opt/gobblin/gobblin-dist/job_work/task-data
data.publisher.final.dir=/opt/gobblin/output
Помогите понять в чем может быть ошибка ?